Kirix Strata Blog

Dates tend to always be tricky things to convert, since there are so many different formats used in different software packages. Today we received a support request about converting Excel dates. This should have been straightforward, as Excel simply provides the number of days since January 1, 1900. So, for example:

20    - January 20, 1900
35981 - July 5, 1998
39341 - September 16, 2007

So, as we saw with the JD Edwards Conversion, we should simply need to use the following formula to convert it:

date(1900,1,1) + fieldname - 1

However, it turns out that Excel has a date bug from its very early days, due to an even earlier date bug in Lotus 1-2-3:

“When Lotus 1-2-3 was first released, the program assumed that the year 1900 was a leap year, even though it actually was not a leap year. This made it easier for the program to handle leap years and caused no harm to almost all date calculations in Lotus 1-2-3.

When Microsoft Multiplan and Microsoft Excel were released, they also assumed that 1900 was a leap year. This assumption allowed Microsoft Multiplan and Microsoft Excel to use the same serial date system used by Lotus 1-2-3 and provide greater compatibility with Lotus 1-2-3. Treating 1900 as a leap year also made it easier for users to move worksheets from one program to the other.”

So, in order to properly convert, we need to account for this extra day. And, we get the final formula:

date(1900,1,1) + fieldname - 2

For further information on dates, please take a look at the other posts we've done on this topic in the past.

In the last post, we talked about Julian dates and Unix Timestamp conversion. We recently received a support request regarding another type of date conversion and, since it might be applicable to others, we thought it would be a good idea to show how you can convert it in Strata. The format is C-YY-DDD, which is used by JD Edwards software: the century after 1900, the year in that century and the Julian date within that year. Here are a few examples:

105031 - January 31, 2005
107263 - September 20, 2007
98129 - May 9, 1998

To convert the dates we first want to get the string into the proper year and then we'll add the Julian date to get the proper day/month (which will also take into account leap years). So, let's take the example “107263″ (we'll assume that it is in Numeric format in your field). Here is the full expression you can use in a calculated field:

date(1900+(field1/1000),1,1)+val(right(str(field1),3))-1

Getting the Year

date(1900+(field/1000),1,1)

The first part of the formula takes the string and divides by 1000, which, when rounded, will provide the first three digits (”107). It then adds 1900 to the 107 to give us “2007″. Then the normal date conversion applies and we end up with “01/01/2007.”

Getting the Day and Month

val(right(str(field1),3))-1

Here we are adding the Julian days for the year to the date we created above “01/01/2007″ — so, if we added “1″ to this date, we would get 01/02/2007, and so on. Here we are using the RIGHT() function to pull out the 3 digits on the right side of the field, in this case “263.” Because it was numeric, we needed to first convert it to a string using the STR() function and then we converted the result back to a numeric value using the VAL() function. This gives us “09/21/2007.” Because we started at “1″ (January 1st) instead of “0″, we simply need to subtract 1 from our date to obtain the correct date of “09/20/2007.”

NOTE: If your JDE date was a character field instead of a numeric field to begin with, you could change the structure or just convert manually in your expression with the VAL() function as follows:

date(1900+(val(field3)/1000),1,1)+val(right(str(val(field3)),3))-1

If anyone has other date conversions they're having trouble with, please let us know and we'll see if we can help.

Converting date formats can be a tricky thing. Two of the more cryptic formats that come up in our support questions relate to Julian dates and Unix timestamps. Here's how you convert them in Strata:

Julian Dates

A Julian date is the number of days since January 1, 4713 BC GMT. To convert a Julian date to the current date, you can simply use DATE(0,0,0) + <number>, where <number> is the Julian date. For for example, if the number appears as “2454917″, the formula in Strata would be:

date(0,0,0) + 2454917 [the result would be March, 26, 2009]

Unix Timestamps

A unix timestamp is the number of seconds since 1970. To convert a unix timestamp to the current date, use DATE(<number>), where <number> is the number of milliseconds since 1970. If the timestamp is given in seconds, multiply by 1000: DATE(<number>*1000). So, for example, if you the number appears as “1237657172″, the formula in Strata would be:

date(1238088021 * 1000) [the result would be March, 26, 2009]

As a side note, sometimes one might mistake a Julian or Unix date for a different custom format. For instance, dates could be stored in the format YYDDD or YYYYDDD where YY/YYYY is the year, and DDD is the number of days since the beginning of that year. So 2001003 would be January 3, 2001

For further information on Strata date conversion, please see this help page.

Strata can work directly with fixed-length or delimited text files. Let's say we need to change the structure of a text-delimited file — say, making a field with smaller to truncate unnecessary values and reduce our files size. We can do this using two methods, either by manually changing our values or by changing the values programmatically prior to import. (As an example, we may have wanted to reduce our web log field width size in this example).

Read the rest of this entry »

We've got another tutorial video which we think you'll find useful. The first thing anyone ever wants to do after installing Strata is to get some of their data in. This video will quickly show you the various ways to get data in and out (hint — it's easier than just using the Import/Export functionality):

Also, for future reference, all of our tutorial videos can now be found in a single location: https://www.kirix.com/strata/portal

Continuing with our video tutorial series, here's one on creating calculations and using formulas. This one briefly looks at how to create calculated fields, but quickly delves into more complex formulas and functions. This one would be worth watching, even if you're an expert — it may give you a few new ideas.

Just wanted to get out a couple more tutorial blogs to everyone before the final page gets published:

Sorting:

Filtering:

Enjoy!

We're in the midst of putting together video tutorials that cover each of Strata's main features. I figure I'd leak some of these ahead of time on this blog, as many may find them useful.

This first video is sort of a basic “rules of the road” for getting around the Strata interface. It's a little over 3 minutes long and covers things like using tabs, panels, various views, and controls in the table view like moving columns and editing.

If you're an expert Strata user, you've probably already got these bases covered. However, there's a good shot that many people will probably pick up at least one thing new.

More rogue tutorial videos to be posted soon…

In part 1 of this series, we looked at the basics of using calculated fields, including referencing fields and using functions.

In this installment, we'll look at how you can use calculated fields to extract data values from a column with “dirty” data. This happens more than one might think: if you've ever "cleaned" a list of contact information or migrated databases, you know that you often have to take a column and break it up into multiple parts. Calculated fields make this task very easy.

Let's look at an example. Suppose we have a table that contains raw contact information smashed together in a single field:

Let's extract each of these parts — first name, last name and email address — from the single column into multiple columns. We'll do this by using a calculated field with a function that parses the string, based on a delimiter (such as a space), and returns the relevant portion of the string. Read the rest of this entry »

Calculated fields are one of the cornerstones of data analysis. It's just one of those tools that you end up using again and again. And, if you can master the functions and formulas and make them do your bidding, you'll be well on your way to becoming a data Jedi. This article begins a multi-part blog series on using calculated fields in your projects.

Calculated fields are nothing more than a way to apply a formula to existing columns or fields. (As a quick aside, spreadsheets use the terms row/column, whereas databases use the terms record/field — there are some technical differences, but for our purposes here, they are very similar). A poor man's calculated field is found in a spreadsheet; you create a formula in a blank cell and then drag the cell down to apply it to the other rows. The more traditional calculated field is found in the database world. Here you create a new field that encapsulates a calculation and then you run it across the entire table.

With Kirix Strata, you get a calculated field like that found in the database world, but with the dynamic/instant properties you get in the spreadsheet world. So, when you create a calculated field in Strata, you get a calculation that applies to the entire data set instantly — whether its 10 rows or 10 million. In addition, these fields behave like regular ol' fields, so you can reference them in other calculated fields or use them in your sorts, filters, groups, relationships, queries and other operations. Of course, the actual data can't be edited like in a regular field, but the underlying formula can be tweaked — with the results showing up in your view in real-time.

Let's create some simple examples to show how calculated fields work. We'll look at the historic stock prices of Apple Computer (to try this yourself, just click the “Download to Spreadsheet” at the bottom of the web page and it will open up into a table in Strata):

Read the rest of this entry »