Need Advice on Datafeeds

[Linda Buquet]

Hi Carsten,

Great to see you again. Thanks SO much for all that info. I can tell you have some pent up posts in you since Revenews has been in the re-design process.

You know whenever you ever feel like talking about industry stuff,
you are always welcome to come visit us and post here!

[Larwee]

Carsten, I'm glad you came here and made your post after I notified you about this thread.

It is good to see you here again.

[cumbrowski]

A few more comments

I dug up a few more references to some of the important stuff I was talking about.

See: Complete Website Templates for Affiliates.
Complete Website Templates for Affiliates.

Just shows the same problem, with different content, but the concept is the same.

I was not kidding about how bad it was for data feeds affiliates back then. I just found this old post where an affiliate describes his personal experience with the Google update and how it affected him.

Google Florida Dance and the effects on me - ABestWeb Affiliate Marketing Forum

I just came across two interesting articles to "Florida" and affiliates
SEO after Google's Florida Update

and data feeds and PHP/MySQL
Affiliate Data Feeds

They are good read, except for the annoying AdSense ads all over the place, but he seems to have written the stuff himself. It's not free to republish stuff so I can not refer to a different site with the same content (talking about replication). I seriously have to think about linking to him from my site.. mmh.

He mentioned this script, which is sold for $39.95

Datafeed-to-MySQL version 2.11
PHP | mill -- Datafeed-to-MySQL v. 2.11 DEMO

Does anybody uses this and could share his experiences and opinions about it? Is it worth it? I mean there are free scripts out there for this kind of stuff, but they are all over the place and not as flexible as needed to become a one stop shop. The $40 would be worth it, if the script delivers what it promises.
Thanks

p.s. Duplication in itsself is not a BIG problem (its still a problem, but the consequences are different), but massive duplication without providing anything unique is.

[cumbrowski]

Originally Posted by Linda Buquet

Hi Carsten,

Great to see you again. Thanks SO much for all that info. I can tell you have some pent up posts in you since Revenews has been in the re-design process.

You know whenever you ever feel like talking about industry stuff,
you are always welcome to come visit us and post here!

Hi Linda,
long time no see, hear, chat, discuss.. whatever hehe

I do have a few pent up posts and ideas flying around, but not in this case. Larwee send me an email and pointed me to the post and I just felt the need to respond to it. I would probably have blogged it and then posted a summary here with a link to it, if ReveNews would have been up

Most posts I do are virtually instantaneous. I am not a news blogger and don't have my own internet marketing blog for that reason. I blog too irregularly.

If the day would just have more than 24 hours, I would come more often here, because I love you guys, but honestly, I think that this would have been better being posted at a blog. It has usually more reach and is easier found down the road to refer to it. I remember my forum posts before I was blogging. Some were actually pretty good (and long) and are now buried between a lot of noise and hard to find. Don't get me wrong, I think a forum is great for things like seeking help and advice for specific problems, but some stuff is better suited for a blog.

Every time a bigger problem came up, did my blog posts turn into essays

I am pretty sure, that you know what I mean, at least I hope that. I don't want to divert from the subject of the thread too much. If you would like to discuss this subject further with me and/or others, feel free to start another tread for it or shoot me an email and we can't talk about this privately.

Cheers!
Carsten

[Linda Buquet]

Yes I totally understand what you are saying Carsten. Funny though for lots of reasons I'm starting to shift away from blogging and putting more focus on posting here more. Forums are still a very important part of social networking and so much more interactive than blogging.

But you're right, this thread is about datafeeds.
Thanks again for posting all your valuable insight!

[cumbrowski]

Btw. Blogs start getting the same problem . People start using social media platforms more and more for things that used to be blogged. Some even asked the questions if blogs are going to die. Nope, they will probably not anytime soon, but stuff where blogs were not the best choice to do it in a first place, will be replaced by things that are better suited for it and blogs will be used for what they are still best for, just like forums.

So, back to datafeeds.


I did not elaborate on the use of web services and the problems with real time data feeds in my first comment and some people who don't know about those kind of things might have a problem with understanding that part of my comment.

APIs (in form of a web service or not), allows to get real-time data for a small set of products or even specific products. In case of CJ, real-time as in the information CJ has for the product at that moment. If the merchant did not update the feed at the CJ network, the information could still be wrong.

Real-time datafeeds as in pulling the feed with current information (and not from whenever a data feed file was the last time provided) still has the problem that you get too much data back and still have to do the full processing on your end.

Since merchant provide the feed though, is the possibility of errors and problems with the feed somewhat smaller, since the network as middle man who is in some case causing the problem, is out of the loop.

If the real-time interface allows filtering of the results down to a very small set of products or even individual products, we start calling it API and not FEED anymore. A simple filter by department/category or type of product is sometimes an option, but still returns more data than you need. This adds to the overhead for transfering and processing the data and makes it often inefficent to use that kind of interface for real time calls on the fly when a user hits a page on your site. That on the other hand causes issues for you and the merchant if you have a large amount of traffic and calls to make.

I hope that clarifies things a bit more

[Silencio]

Carsten,

Thanks for stopping by and sharing about this topic. I appreciate it very much and I am sure that others will too.

While I am just getting my feet wet with Datafeeds I am learning quickly that this powerful media and marketing technique is very lacking in many ways!

I am not a programmer and have only dabbled enough to know that is another profession....that I am not going into, but what I do know about it is enough to know that I need help with it.

While my first attempt at getting datafeeds on my pages indeed did use the program RSS2Blog to post it to a Wordpress Blog this program and Wordpress also can post to regular html pages. I am sure you are aware of that also as Wordpress has really evolved into a powerful CMS software instead of just a Blogging software.

So in my first attempt, with the available information that I had available, was somewhat successful in that I did indeed get datafeeds uploaded to my page.....but that was the start of my questions....that brought me here to ask questions.

While this first attempt I consider a minor major accomplish for me in my goals to learn datafeeds I now realize that this was just a baby step in the right direction.

I have had web pages since 1995 and have been building my own since 1999 and I have made it a big effort to learn as much as I can about SEO so while I would not consider myself an expert I am literate of the topic and can hold my own to a certain degree.

I was also dropped off the charts in the Google Florida Update but it was not because of a duplicate content penalty as all of my content on that main page of mine was original and created by me. I think a lot of other factors were involved. And in today's world there is a lot of duplicate content posted on web pages especially in the form of RSS feeds and it doesn't seem to affect them to the extent that maybe it did, maybe because of the rise of RSS since the Florida Update. Just an observation.

After TheCaymanHost posted your web page (Thanks again) I have spent quite a bit of time there and you have a tremendous amount of information there to digest.....and explore and it looks and feels like I will find lots of answers there but....

in my searches so far I have not found a good guide that says "If you want to get started with Datafeeds this is the best way to it from the Beginning" which is why I have been going around the block in this learning adventure.

What I find is "You could do it this way" or "You could do it that way" or "Maybe this way".........guess I chose a way that might not work as well as others although I have this strong feeling that if a way is found that would allow one to update feeds easily in a WordPress application it could out perform some other options.

I see a huge need for a revolution in the way datafeeds are produced and distributed and I know that you have been dealing with this issue much longer than I. Is there a reason that datafeeds are not more standardized?

Carsten you advice is much appreciated! Thanks. I hope that through your help and resources I can find a good solution or two or three.....

But let me ask the question as I end this post.....

If you, with the knowledge that you have today, were to recommend to someone the best way to get started with Datafeeds, to get them started in the right direction, what would you suggest to them in a step by step approach?

Thanks

Charlie

[Silencio]

Originally Posted by cumbrowski

He mentioned this script, which is sold for $39.95

Datafeed-to-MySQL version 2.11
PHP | mill -- Datafeed-to-MySQL v. 2.11 DEMO

Does anybody uses this and could share his experiences and opinions about it? Is it worth it? I mean there are free scripts out there for this kind of stuff, but they are all over the place and not as flexible as needed to become a one stop shop. The $40 would be worth it, if the script delivers what it promises.
Thanks

Carsten I also found this post on another forum where someone posted a free script that will upload to an MySQL database.


(If this outside link is not allowed to be posted here please remove it and accept my apologies but as were were discussing this I thought it might be of value...)

And I have a question for you on this Datafeed to MySQL approach if you could please help me understand the importance of this technique.

If you have uploaded datafeeds directly to your MySQL database today and next week you need to Update your datafeed, how is this accomplished?

And also in relation to my site since a datafeed in a WordPress site is also input into a MySQL database is there an easy way to update a feed by uploading directly to your MySQl database?

Charlie

[cumbrowski]

Hi Charlie,

Originally Posted by Silencio

And in today's world there is a lot of duplicate content posted on web pages especially in the form of RSS feeds and it doesn't seem to affect them to the extent that maybe it did, maybe because of the rise of RSS since the Florida Update. Just an observation.

If you are interested in the whole duplicate content thingy, have a look here. There is no simple answer to that one either .

Duplicate Content and Near Duplicate Content Issues -Resources for SEO

Originally Posted by Silencio

in my searches so far I have not found a good guide that says "If you want to get started with Datafeeds this is the best way to it from the Beginning" which is why I have been going around the block in this learning adventure.

The thing is that the subject is for beginners and requires serious technical knowledge to do it right. There are some scripts but they either produce something that is pretty much worthless or you are required to customize it so extensively that it is not easy again. But I see your point, its not spelled out very often. I am thinking about taking some of my responses here and craft something out of it to put on my site.

Originally Posted by Silencio

I see a huge need for a revolution in the way datafeeds are produced and distributed and I know that you have been dealing with this issue much longer than I.

Amen to that. I can point you to forum posts that are 5 years old where you can hear me ***** and yell (after I made some very specific and relevant suggestions). Regarding the why. Not sure, not a priority with the networks, obviously.

Originally Posted by Silencio

Carsten you advice is much appreciated! Thanks. I hope that through your help and resources I can find a good solution or two or three.....

My pleasure, I am glad that I could help.

Originally Posted by Silencio

If you, with the knowledge that you have today, were to recommend to someone the best way to get started with Datafeeds, to get them started in the right direction, what would you suggest to them in a step by step approach?

- Get at least some programming skills and also some experience with database software (MySQL or MS SQL Server (which I use))
- Get familiar with delimited text files and their problems. You can use Excels import feature to import feeds. There you will see fairly quickly what some of the issues are
- never assume that the feed file is okay and that the columns are used for what they are supposed to be used
- Get familiar with FTP (basic FTP that comes with Windows is all you need)
- Get familiar with XMLHTTP to pull data via HTTP
- Get familiar with data validation and data normalization principles and problems
- Learn how to schedule things for automation. I use SQL Server Agent to schedule things, but the Windows scheduler would work for a lot of things too
- Basic XML knowledge would not hurt either
- Get familiar with web services and use them whenever you can.
- Amazon's and eBay's APIs are good for learning and getting experiences with web services


If this is beyond you, have a closer look at the 3rd party services that do all this work for you and provide product data in an easy to use format.

Originally Posted by Silencio

Carsten I also found this post on another forum where someone posted a free script that will upload to an MySQL database.

That one is for the ShareASale data feeds and not generic, although it could probably be modified to use it for other feeds as well, but you need some PHP knowledge to be able to do that.

Originally Posted by Silencio

(If this outside link is not allowed to be posted here please remove it and accept my apologies but as were were discussing this I thought it might be of value...)

It is okay, because it is relevant to the discussion

Originally Posted by Silencio

If you have uploaded datafeeds directly to your MySQL database today and next week you need to Update your datafeed, how is this accomplished?

Two options

You receive in almost every case a full product feed.

1) Delete all records related to the feed in the database and then import it as if it was new. Depending on the feed is the key for the deletion filter usually the merchant ID on the network where you pull the feed from. The problem is that you can't use your own primary key for products in this case, because the key values would change and all your references to the old records would become invalid

2) Check if the product exists in the DB by doing a look-up with a unique identifier for the Merchant+Product combination and do an update if it exists (have a date stamp for "last updated" and update that as well). Insert all records you can't find in the DB. Delete all records where the "last updated" stamp was not updated. This requires more logic for the import script and takes longer to import data.

Hope that helps
Cheers

[TheCaymanHost]

Phew - I'm certainly glad I pulled Carsten in on that one

I realized that the original question was not really answered at your site, but you still have one of the most comprehensive and understandable resources out there on the whole subject.

Sorry I created so much work for you! Thanks for helping out, and clarifying so much in your posts - I'm sure this thread will now be a great help to many forum visitors.

Mo