Maffo Bulk Upload System

[Maffo]

Faster than a speeding bullet (hmmm.... maybe not that fast but certainly faster than Ben Johnson in 1988)
More powerful than a locomotive (I would have to say yes)
Leaps tall buildings in a single bound (havent tested this one yet)

This my friends is the Maffo Bulk Upload System.

I have been busy developing a bulk uploading system to run on 68 classifieds 3.1

Having only just completed the system and ran some benchmark tests, I have to say that it is nothing short of awesome and by far my best programming yet.

What it does:
Allows you to upload literally thousands of listings to your website via a CSV.file
A csv file is basically information from a spreadsheet or database saved in a text format.

This system processes your images, inserts them into your photo folder, resizes them if necessary and also makes the thumbnails for them.

The most difficult part about writing this system was the knowledge that everybody has a different system with different categories, amount of images,extra fields, listing packages etc. I have done my utmost to make sure, and I am 100%, confident that it will cater for everybodies needs.

This system has been tested with ZERO server timeouts.

A benchmark test I ran on a fairly standard, shared server running Apache/linux gave me some really optimistic results.

5000 listings, each with 2 images of dimensions 800*600px (avg 100kb) set to be reduced to 640*480.

Thats 10000 images to be processed and also set to a thumbnails

Each listing also has 2 extra fields, a short description, and a full description

5000 listings took 76mins 32 secs

Yes that might seem like a hell of a long time in computer terms but when you we look at the figures we can see that its taking less than a second to..

• Pull the record from the database
• xss clean everything!!
• strip any unallowed html
• Update your orders table
• Update your Products Table
• Add two extra field entries
• Resize two large images and also create their thumbnails

Strip out the images and the system can fly like a knife through butter.

I have also written a delete feature which allows me to remove everything from that last import including the images from the image folders. This is incase I upload my csv only afterwards to realise I have placed something wrong. (even though there are lots of checks before the actual processing of the csv, I felt this was a necessary feature)

This benchmarked at under 3mins which is pretty damn good

I then took the same csv sheet, removed the 10000 images and submitted 5000 plain text ads to the upload system. This also came in at under 3mins so I clearly tell that is the image manipulation that slows this whole process down.

There is a category validator which compares what is on your CSV to what is allowed in your db (works like a dream and saves you having to manually type out any categories)

There is an Image validator which checks that the images in your CSV file actually exist and if not allows you to rectify things before continuing.

Prepping the information can be the most time consuming so I have made it possible to back up your work at any time so you can continue at a further time.

Like I said earlier this is by far the finest work I have done so far, and has pushed new limits for 68 users.

For those with property websites, there is now the ability to contact hundreds of estate agents and get them to send you their properties, the same with cars, boats whatever?

On testing, it seemed strange seeing my empty site suddenly with 5000 ads?? and best of all, how long would it have taken me to put them all in manually??? I dread to think, at lets say 2 mins an ad? Thats a whole lotta man hours!! 166 to be precise and thats without a coffee or cigarette break!!
Any questions just fire away. Apart from how much it costs. I really dont know yet but it isnt going to be peanuts. I really feel this has its value.

[manilaboy]

Originally Posted by Maffo

Faster than a speeding bullet (hmmm.... maybe not that fast but certainly faster than Ben Johnson in 1988)
More powerful than a locomotive (I would have to say yes)
Leaps tall buildings in a single bound (havent tested this one yet)

This my friends is the Maffo Bulk Upload System.

I have been busy developing a bulk uploading system to run on 68 classifieds 3.1

Having only just completed the system and ran some benchmark tests, I have to say that it is nothing short of awesome and by far my best programming yet.

What it does:
Allows you to upload literally thousands of listings to your website via a CSV.file
A csv file is basically information from a spreadsheet or database saved in a text format.

This system processes your images, inserts them into your photo folder, resizes them if necessary and also makes the thumbnails for them.

The most difficult part about writing this system was the knowledge that everybody has a different system with different categories, amount of images,extra fields, listing packages etc. I have done my utmost to make sure, and I am 100%, confident that it will cater for everybodies needs.

This system has been tested with ZERO server timeouts.

A benchmark test I ran on a fairly standard, shared server running Apache/linux gave me some really optimistic results.

5000 listings, each with 2 images of dimensions 800*600px (avg 100kb) set to be reduced to 640*480.

Thats 10000 images to be processed and also set to a thumbnails

Each listing also has 2 extra fields, a short description, and a full description

5000 listings took 76mins 32 secs

Yes that might seem like a hell of a long time in computer terms but when you we look at the figures we can see that its taking less than a second to..

• Pull the record from the database
• xss clean everything!!
• strip any unallowed html
• Update your orders table
• Update your Products Table
• Add two extra field entries
• Resize two large images and also create their thumbnails

Strip out the images and the system can fly like a knife through butter.

I have also written a delete feature which allows me to remove everything from that last import including the images from the image folders. This is incase I upload my csv only afterwards to realise I have placed something wrong. (even though there are lots of checks before the actual processing of the csv, I felt this was a necessary feature)

This benchmarked at under 3mins which is pretty damn good

I then took the same csv sheet, removed the 10000 images and submitted 5000 plain text ads to the upload system. This also came in at under 3mins so I clearly tell that is the image manipulation that slows this whole process down.

There is a category validator which compares what is on your CSV to what is allowed in your db (works like a dream and saves you having to manually type out any categories)

There is an Image validator which checks that the images in your CSV file actually exist and if not allows you to rectify things before continuing.

Prepping the information can be the most time consuming so I have made it possible to back up your work at any time so you can continue at a further time.

Like I said earlier this is by far the finest work I have done so far, and has pushed new limits for 68 users.

For those with property websites, there is now the ability to contact hundreds of estate agents and get them to send you their properties, the same with cars, boats whatever?

On testing, it seemed strange seeing my empty site suddenly with 5000 ads?? and best of all, how long would it have taken me to put them all in manually??? I dread to think, at lets say 2 mins an ad? Thats a whole lotta man hours!! 166 to be precise and thats without a coffee or cigarette break!!

Any questions just fire away. Apart from how much it costs. I really dont yet but it isnt going to be peanuts. I really feel this has its value.

This is awesome! How does it handle billing?

[Greg H]

I am the one who requested Maffo develop this and it was not one of my requirements for it to handle billing.

Maffo is the only one that can say if/how it handles billing, but the clients of mine for who this will be used are a select few large clients and the whole deal/payment for it will be handled offsite. I imagine there will be many variables in regards to how much work and cost involved for individual clients.

I haven't seen the product yet, but it I'm getting very excited about it.

Some of my clients will be adding thousands and thousands of listings using this, so it will be put the test pretty quickly and I have much confidence in Maffos "Bulk Upload System".
I have to because without it, my site would end up being just another site with just a few listings.


Cheers

[VanWertAds*com]

I would be interested in this mod... whats the price?

[Maffo]

Any questions just fire away. Apart from how much it costs. I really dont know yet but it isnt going to be peanuts. I really feel this has its value.

I dont have that answer yet Im sorry vanwart.

This is awesome! How does it handle billing?

It cant Manilla. The idea of this system is you can contact a client, lets say a real estate agency. You can ask them for all of their properties in the form of a spreadsheet plus a copy of all their images. YOU have to upload their properties, NOT the client. Allowing somebody to possibly inject thousands of unchecked records into your system would be suicidal.
What I suggest is you dont use the 68 billing system for this, you can simply agree a price with the client and send them a bill via payapl or some other form of payment, cash if your lucky!

I am at the moment, writing the guidelines on how to prep the information, ready to be inserted into your database properly so we dont have any little accidents.

I really am excited by this system, Im in two minds whether to pack in the programming and focus more on marketing, as I know I now have the tools to approach clients and get literally thousands of ads for content.

BTW Vanwart, I have my own version of your signature.

Do unto others as others do to you. Its almost the same but very different.

[racingjon]

Hope you plan on releasing this for V4 once it's out of beta!

[Maffo]

As long as the structure of the database hasnt altered too much it should be able to do so without much hassle.

There is one thing Im trying to do which validate extra fields but its a complete an utter minefield.

Once this is achieved I will look at getting it working for v4

[Maffo]

Project complete!!!

This project is now complete and ready for sale.

I am setting a price of �150GBP on this product and is on a per domain basis license. I am open for negotiations on use for multiple domains.

The reason for the high price is because quite simply, I don't want every Tom, Dick & Harry having this privelage. I want MY clients to have a huge advantage over their competitors. It is a fair amount to pay, but to a serious business, this is nothing. No disrespect but I am not interested in penny websites and I have no doubts that to the right business, this outlay can be recouped within a few days. To list 1500 ads on your site at 10pence a listing, you have broke even or 300 ads at 50pence ($1 dollar).

You now have it in your power to contact business with literally hundreds or thousands of products they offer such as properties, autos or absolutely anything else.

Your website will be one full of content, there is nothing worse than an empty website.

For those of you who need have modded your site, we can make arrangements to modify the system to suit your needs.

You can place an order by sending �150GBP to maffo at maffo dot co dot uk

I have written a fully comprehensive manual to help you all through the usage of the system. I have placed this online if you wish to have a look.

Maffo Bulk Upload Readme

[Maffo]

Well I am back in the office so as promised the price of the Bulk Upload system will now increase to its proper retail value of �150

Many thanks to those who took advantage of the offer. I will be sending you your product immediatley.

[eagle6]

YOU have to upload their properties, NOT the client. Allowing somebody to possibly inject thousands of unchecked records into your system would be suicidal.

The project we are planning would have at least 200 (hopefully around 500) accounts that would need to upload a file of between 5 and 50 records at least once a week. Is there any way we can allow account holders access to upload their own batches? ...... or are we looking at the wrong software (68 Classifieds)?

Your expertise in this matter would be appreciated.
TIA