Unfortunately, none of the three CSV formats that Excel outputs could be properly parsed as CSV. For example, accepted CSV standards state that values should be:

"value1","value2","escaped "" double quote", "value 3", "", "", "7th column"

But in Excel’s version of CSV, that would look like:

value1,value2,"escaped "" double quote",value 3,,,7th column

It’s terribly inconsistent in its use of double quotes, confusing the hell out of many CSV parsers.

The Solution:

CSV to XML Converter by Creativyst(R). It takes in CSV garbage and returns immaculate and clean XML. Then use your language-of-choice’s XML functions to work with the data.

Let’s start at the beginning, at installation.  Your web server likely has capable hardware, so you need to be sure that your software configuration meets requirements.  Luckily the host I’m on was set up just fine the way it was.  Obviously you need to download the software – I chose to download the Full Release (version 1.1.8 at the time).  The zipped package contains 6k+ files so you may want to look into an “unzip on the server” solution or you’ll have to wait a long time like I did.  If you decide to wait, take the time to read through the User Guide and the Designer Guide.

Setting up Magento

Once you have the files uploaded, CHMOD the directory (where you dropped those files) to 777.  Then navigate to the installer script – http://www.yourdomain.com/your-magento-folder/.  The script is very simple, and the most important part of it is inputting database connection information.  Proceed to create an admin account and log in to the administration panel.

I easily set up a “website”, “store”, and “store view”.  For my purposes, just one of each was necessary so I named them “clientname_website”, “clientname_store”, and “clientname_storeview”.  I haven’t set up any roles other than Administrators yet, since for now it’s just myself and another developer .

My two main tasks to work on:

1. Style the front-end to look like our client’s existing storefront
2. Import the client’s existing product catalog data (including manufacturer and category data)

I decided to work on styling the front-end first, since it seemed to be the more difficult task.  I imagined thousands of CSS declarations I’d have to modify, not to mention figuring out the XML-based layout system.  As for the catalog import, I imagined it would simply be the mapping of data fields into the Magento model.  I prefer to tackle more difficult tasks first, for peace-of-mind reasons.

Theming Magento

As I predicted, getting Magento to look the way I needed proved to be a very difficult task.  I was hoping for a more Wordpress style header/content/left+right/footer type of deal, but what I saw was something much more complicated.  To make the simplest of changes took many hours, with my development process being something like:

1. Figure out which file(s) contains the code (HTML, XML)  I need to edit.
2. Verify I have the right files.  Make a small test change and see if output is affected.
3. Remind self of what “block, layout, and template” mean.
4. ( trial/error, server timeout ) x 10
5. Break.  Resume. Now, what was I doing? /MyCompany/MyCompany/?  Oh right.
6. Search Google on how to solve the problem.  Search is fruitless.  Try Experts-Exchange.
7. ( trial/error, server timeout ) x 10
8. Almost there.
9. Major revelation! Break.

All in all, theming was/is very complicated.  It’s hard to determine what references what, and how to go about making changes to get the desired effect.  My impression is that the Magento team would have us use their system in one way, and only that way.  The whole hierarchical system seems like a good idea, but in practice it’s cumbersome and difficult to master.

Importing the Product Catalog

After walking barefoot over hot coals, or theming Magento, I was really hoping that importing the product catalog would be a no-brainer – just map fields in my existing database to fields in the Magento database.  I couldn’t have been more wrong.  Magento’s database model is an instance of the entity-attribute-value model (EAV).  The difference between EAV and conventional modeling is that table records don’t represent an instance of a data object.  Instead, the data object’s values are spread out over multiple tables, making manipulating the delicate database directly incredibly complex.

The route I decided to take, as recommended on several sites, was to:

1. Create a single category (under “root category” or “Default Category” of course)
2. Create and assign a single product to that category.
3. Run the “Export All Products” profile to generate a CSV file
4. Export existing data to CSV according to the exported CSV’s column format
5. Upload the new CSV using the “Import All Products” profile.

When exporting existing data to the Magento format, be sure to use the category ID values for category instead of the category names.  You can find out a category’s ID by navigating to its “edit category” page and looking at the URL.  The ID is in the URL.

When running the “Import All Products” profile, you might have to limit the number of records that you import at a time.  If that doesn’t work, (as in my case) try reducing the number of rows that are in the CSV file.  You would have to upload many many CSV files and run the profile over and over again for those files.  I don’t understand why adding records to just a few tables should consume a large share of server resources.  Import crashes for me a lot, which isn’t good because I’m unclear of whether all products have been imported or not.

Speed and Performance

I’ve decided to dedicate a section to speed and performance, because Magento’s slowness was a huge distraction during the development process.  It would be nice if at the very least that the system didn’t crash every 1/5 page loads.  It would also be nice if I could see the results of a few CSS declarations quickly.  There seems to be a LOT of overhead in Magento, I could go on forever about it.

Based on what I’ve read, you should go with a dedicated or a virtual-private-server (VPS) if you choose to run Magento.  A lot of developers working on a shared hosting solution seem to have huge speed issues.  Trust me, save yourself a lot of headaches and upgrade your hosting.

I had no experience whatsoever with interfacing Flash with the server. I was surprised to learn that the task wouldn’t be as difficult as I’d imagined. An SWF file can be set to load data from a server transparently, akin to AJAX’s behavior. The only hint of an HTTP request is seen in the browser’s status bar. Anyway, an SWF can query an XML file (or in my project’s case, XML-output generated via PHP) and parse that into objects that can be displayed to the user.

Whenever the SWF has to display data in a pre-defined format, for example for a biography page, I can simply send it xml in the format:

<person>
<name>Person Name</name>
<birth>20000101</birth>
<biography>Born.  Lived.  Died.</biography>
</person>

Nice and simple, the data required by the SWF can be easily represented through XML structures.

When the data is not so structured however, we can feed it into flash via XML’s <![CDATA[content here]]> value. It’s good to know that Flash can display HTML content, albeit in limited fashion. For example, CSS is not readily supported (I believe it can be done using ActionScript, and that requires work on the Flash programmer’s part). Since there’s no CSS, we have to use the “old-school” <b>, <i>, and <u> tags, as well as the dreaded <font> tag. Remember <font>? Those were the good old days. For the official reference on supported HTML tags, here’s Adobe’s “Supported HTML tags” page. Enjoy.

It’s important to note also, that when using the <b> and <i> tags, that the specified font has the bold and italic typefaces. One of the issues I ran into was that for some reason the SWF wasn’t honoring <b> and <i> tags, due to this typeface unavailability.

[Edit: updated link to Supported HTML tags]