cctld Technical help regarding google bot

[DropWizard.com]

Google bot has been pounding a couple of my ca sites this month. 29k visits on one and 12k on the other blowing through about 1.2 gigs of bandwith for sites that were set for 250mb. I've had to raise the bandwidth a couple of times to keep them up. I'm not sure what solution can be done to this problem other than banning their IP but that could have detrimental effects on rankings.

Is there a solution to this? Or??????

 

[Tia Wood]

Do you mind posting the URL?

 

[DropWizard.com]

A link is a link

http://www.inkrefill.ca

http://www.digitalprojectorbulb.com both are using associate-o-matic amazon stores.

 

[south]

If you have the site verified with google, you can set a custom crawl rate for it in google's webmaster tools.

Find it under site configuration / settings.

https://www.google.com/webmasters/tools/

 

[Tia Wood]

In addition to south, I think they are getting stuck on the pagination numbers in which you can add nofollow links to if associate-o-matic allows customizing like that. But I definitely try south's suggestion first.

 

[south]

And I would personally be careful about the adsense on ink refills. Not sure if that fits in their tos or not. I suppose it depends on how they count ads. But with no content, could get you banned, or the site blocked. Again, not sure, just my initial thoughts.

 

[DropWizard.com]

Thanks south I hadn't thought of the google webmaster options.

The software allows the input of the adsense code and I've seen other sites using adsense also. But they do have a forum so I'll follow that thought up. Not that I much care we're removing adsense from all our sites as they get revised. I'm tired of getting thrown pennies on the dollar while they take the majority of the rev.

Tia

thanks for the suggestion. I've pm'd you some info.

 

[Tia Wood]

Thanks. PMed you back.

 

[POLiSH]

I would not ban the bot but rather adjust the crawl rate using the google.com/webmaster tools for the domain in question.




As stated above ^^^

 

[whitebark]

Many spam bot operators fake the user agent and claim it as google bot. Check the ips - if not google's - ban them.

 

[Tia Wood]

Using google.com/webmaster tools or adding a nofollow might be ignored by other bots. There may be another solution as his pagination is set to 400 pages max, 10 products per page and under multiple categories, that adds up fast. There is a feature in associate-o-matic that I directed him to that would lower the pages it returns. That is where they are getting stuck. Resolve that and there is no need to ban, adjust or limit anything unless he didn't want it crawled at all.

 

[POLiSH]

Using google.com/webmaster tools or adding a nofollow might be ignored by other bots. There may be another solution as his pagination is set to 400 pages max, 10 products per page and under multiple categories, that adds up fast. There is a feature in associate-o-matic that I directed him to that would lower the pages it returns. That is where they are getting stuck. Resolve that and there is no need to ban, adjust or limit anything unless he didn't want it crawled at all.

Good points, for some reason I was thinking google was the only one in question. Going back to his first post. We are not talking the avg site, he's looking at almost 30,000 visitors per month, so this is a high traffic site with lot's of images. Let's look at the size of the page.

The Ink Refill site.

44699 kb x 30,000 = 1,340970,000. and that's just for one page.

If you think that 250meg will hold 30k in visitors per month on a ecommerce store, then I am missing something.

 

[DropWizard.com]

Good points, for some reason I was thinking google was the only one in question. Going back to his first post. We are not talking the avg site, he's looking at almost 30,000 visitors per month, so this is a high traffic site with lot's of images. Let's look at the size of the page.

The Ink Refill site.

44699 kb x 30,000 = 1,340970,000. and that's just for one page.

If you think that 250meg will hold 30k in visitors per month on a ecommerce store, then I am missing something.

Perhaps the fact that it gets around 20 visitors/day. The balance of the bandwidth is being eaten up by googlebot. I've reduced the 400 to 100 and the 100 to 50 as you suggested Tia. We'll see what affect this has. I went to their support forum and frankly there wasn't much help although there are others with the same problems.

I've also noticed the growth in disk space used and wonder if it's a result of googlebot hammering all those pages that are set for caching. Perhaps a clearing???

 

[DropWizard.com]

Ok a sample of recent visitors: As you can see it appears googlebot is trying to index amazon via our site All coming from 66.249.71.82
A lot of this stuff doesn't even belong on our site: (digital bulb btw)


/index.php?c=LCDProjectors&n=160356011&i=B0001LTT4Q&x=HP_48X_CDR_100_SPINDLE
/index.php?c=DigitalProjectors&n=160355011&i=B00004Z6N6&x=Belkin_36_Piece_Demagnatized_Computer_Tool_Kit_with_Case_Black
/index.php?c=LCDProjectors&n=160356011&i=B00011R2VS&x=Sony_DRU_530A_8x_Internal_Dual_Format_DVD+_RW_Drive
/index.php?c=DigitalProjectors&n=160355011&i=B00004Z6XS&x=Belkin_F8E062_55_Piece_Computer_Tool_Kit_with_Black_Case_Demagnetized_Tools
/index.php?c=LCDProjectors&n=160356011&i=B0002340SI&x=KHypermedia_25_x_DVD+R_47_GB_4x_spindle_storage_media
/index.php?c=DigitalProjectors&n=160355011&i=B0002W3A6W&x=Paladin_Tools_1576_Data_Link_ID_And_Cable_Check
/index.php?c=LCDProjectors&n=160356011&i=B000233WGO&x=Khypermedia_Spindle_of_25_Blank_4x_DVD_R_Discs
/index.php?c=Home&n=281056&i=B000RZPK1M&x=Cyberpower_CP825LCD_UPS_825VA_450W_LCD_Display_8_Outlet_RJ11_Compact_Design_EMI_RFI
/index.php?c=VideoProjector&n=160355011&i=B000ON2TQG&x=Panasonic_Prestige_NN_SD997S_22cuft_1250_Watt_Sensor_Microwave_Oven_Stainless_Steel
/index.php?c=LCDProjectors&n=160356011&i=B0043862N4&x=Mini_Bluetooth_Wireless_Keyboard_for_iPad_iPhone_40_OS_Android_Window_Mobile_Symbian_Smartphone
/index.php?c=VideoProjector&n=160355011&i=027373279X&x=The_New_Business_Road_Test_What_entrepreneurs_and_executives_should_do_before_writing_a_business_plan_3rd_Edition
/index.php?c=VideoProjector&n=160355011&i=B000I98ZM8&x=2_x_RCA_Female_1_x_35mm_Mono_Female_Adapter
/index.php?c=DigitalProjectors&n=160355011&i=B000NBCEM8&x=Sylvania_SA135_10_Amp_Table_Top_Timer_with_Motion_Sensor_and_Alarm
/index.php?c=DigitalProjectors&n=160355011&i=B000053U3X&x=Star_Wars_The_Force_Unleashed_Ultimate_Sith_Edition
/index.php?c=Home&n=281056&i=B000FQ2D5E&x=Metal_Gear_Solid_4_Guns_of_the_Patriots
/index.php?c=VideoProjector&n=160355011&i=B004DJSVIW&x=X_Mini_II_Capsule_Speaker_Pink
/index.php?c=DigitalProjectors&n=160355011&i=B00009N6VJ&x=Monster_SV100_F_25_WHT_Coaxial_Video_Cable_25_feet
/index.php?c=DigitalProjectors&n=160355011&i=B000O8AHLA&x=GLS_Audio_Locking_Series_Generation_4_Gold_Connector_Banana_Plugs_Banana_Clips_20_Pack_10_Reds_10_Blacks
/index.php?c=DigitalProjectors&n=160355011&i=B00004Z61J&x=Belkin_18_Piece_Computer_Tool_Kit_with_Demagnetized_Tools_Black_Case
/index.php?c=LCDProjectors&n=160356011&i=B004Z75V08&x=Contour_1500_Contour_Plus_Camera
/index.php?c=DigitalProjectors&n=160355011&i=B00328HR5S&x=Sony_DSC_W330_141MP_Digital_Camera_with_4x_Wide_Angle_Zoom_with_Digital_Steady_Shot_Image_Stabilization_and_30_inch_LCD_Blue
/index.php?c=3D&n=160356011&i=B0021AEPTY&x=Elgato_10020196_Turbo264_HD_Encoder_Accelerator
/index.php?c=node&n=3349991&i=B000QDIUEO&x=JVCC_ART_11_Artist_Board_Console_Tape_3_4_in_x_60_yds_White
/index.php?c=node&n=3349991&i=B00009XVIS&x=Photoflex_FV_Connector_7
/index.php?c=LCDProjectors&n=160356011&i=B004GHMS44&x=Seidio_Ultimate_Screen_Guard_for_Samsung_Focus_2_Pack_Crystal_Clear
/index.php?c=LCDProjectors&n=160356011&i=B000E8BGCE&x=Ethernet_Cable_CAT5e_25_ft_White
/index.php?c=LCDProjectors&n=160356011&i=B0046HNWO4&x=3M_CP40_Shootn_Share_Camcorder_with_Projector_White
/index.php?c=LCDProjectors&n=160356011&i=B0029NRM2U&x=Magic_Flight_Launch_Box
/index.php?c=VideoProjector&n=160355011&i=B004LU1CA2&x=Satechi_BT_Wireless_Bluetooth_Portable_Speaker_System_for_iPhone_Android_Smart_Phones_iPad_Tablets_Macbook_Notebooks
/index.php?c=LCDProjectors&n=160356011&i=B0002EH4YO&x=Mini_Multi_Battery_Tester
/index.php?c=DigitalProjectors&n=160355011&i=B00386VMWI&x=NCAA_Football_11
/index.php?c=LCDProjectors&n=160356011&i=B003O6N64Y&x=Ear_Force_PX5_Programmable_Wireless_71_Dolby_Digital_Surround_Sound_Headset_with_Bluetooth
/index.php?c=3D&n=160356011&i=B000HJPK2C&x=Nikon_70_300mm_f_45_56G_ED_IF_AF_S_VR_Nikkor_Zoom_Lens_for_Nikon_Digital_SLR_Cameras
/index.php?c=VideoProjector&n=160355011&i=B000YZAZMO&x=Evenflo_ABC_SmartSteps_ExerSaucer
/index.php?c=LCDProjectors&n=160356011&i=0321749847&x=Apple_Pro_Video_Training_Aperture_3_Apple_Pro_Training
/index.php?c=3D&n=160356011&i=B003UATR3I&x=Elgato_EyeTV_Hybrid_10021020
/index.php?c=Home&n=281056&i=B003EQ4C92&x=Royal_Opulance_Satin_Queen_Sheet_Set_Giraffe_Brown_Ivory
/index.php?c=LCDProjectors&n=160356011&i=B004LDE3PU&x=Justin_Bieber_Limited_Edition_3_d_Glasses
/index.php?c=VideoProjector&n=160355011&i=B004O77NLE&x=Samsung_UN40D6500_40_Inch_1080p_120HZ_3D_LED_TV_Black
/index.php?c=LCDProjectors&n=160356011&i=B00009W3DU&x=RCA_DLNB2BOX_LNB_Dual

 

[Tia Wood]

I have no clue how that's happening. I followed a few links and it sends me back to the first page as if they weren't found. Google isn't going to crawl Amazon via your site and it isn't going to modify and follow unSEO friendly links like that. I highly suggest contacting associate-o-matic to find out what could be going on unless POLiSH has an idea of what could be happening.

 

[POLiSH]

From what I have read regarding associate-o-matic and what you are viewing DropWizard, this seems to be a known problem for people using associate-o-matic using Amazon Affiliate program and all of the sudden getting blown away with bandwidth usage. If you have already tuned down the crawl rate on google, give it around 72 hours and see if you see a decrease. If you don't I would be looking for a throttle down option via your amazon account.

 

[DropWizard.com]

Yeah some of the links resolve to pages some don't. And the links are unfriendly that was one of the points I made about having to rewrite the htaccess rules but that's beyond my tech ability.

Well on to associate to see what else. Their solution also seems to be turning down the crawl rate.

Conclusion. Software is not ready for the big time :-(

Thanks everyone for the help. It's greatly appreciated.