tag:blogger.com,1999:blog-534724342082307142.comments2023-02-22T18:30:17.187+02:00Cloud Developer TipsShlomohttp://www.blogger.com/profile/10469902663120418195noreply@blogger.comBlogger82125tag:blogger.com,1999:blog-534724342082307142.post-21512920641987307602010-02-01T10:27:38.033+02:002010-02-01T10:27:38.033+02:00@slitz,
You don't need a premium account to s...@slitz,<br /><br />You don't need a premium account to send email via Gmail using code. Here's a link to some examples for how to do it in Java:<br /><br />http://stackoverflow.com/questions/46663/how-do-you-send-email-from-a-java-app-using-gmailShlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-63880589418977660922010-01-31T03:22:17.190+02:002010-01-31T03:22:17.190+02:00hello,
Can you point out where can I read about u...hello,<br /><br />Can you point out where can I read about using Google Apps email to send email? I though I needed a Premium account for that...<br /><br />Thank you!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-18908995256242587162010-01-25T21:46:39.751+02:002010-01-25T21:46:39.751+02:00We've been experiencing this very issue recent...We've been experiencing this very issue recently and I'm glad to find an article that corroborates my theory. Thanks for posting, Shlomo.Drewhttps://www.blogger.com/profile/06824635985094851696noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-49436004578719872702010-01-07T19:04:12.143+02:002010-01-07T19:04:12.143+02:00@rob- wondering whether you have had success with ...@rob- wondering whether you have had success with the above setup. I have setup my MTA and my clients as well, however for some reason I am still getting a relay access denied. I believe I am having an issue with the certificates on the client side. I'd be most interested in your experiences.<br /><br />Cheers!<br /><br />Michael Klatsky<br />TnR Global, LLCAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-4012903106477551332009-12-27T23:07:57.733+02:002009-12-27T23:07:57.733+02:00@Ramchandra,
You ask interesting questions.
1. T...@Ramchandra,<br /><br />You ask interesting questions.<br /><br />1. There is no way to request a specific subnet for your instance's private IP.<br /><br />2. It's normal for traffic between instances to experience fluctuation, especially when measuring ping time: The EC2 network de-prioritizes ICMP communications (as per the comment from Cindy@AWS in this thread: http://developer.amazonwebservices.com/connect/message.jspa?messageID=156301 ). Note that the SLA does not guarantee a minimum bandwidth or network latency.<br /><br />3. First we need to determine if network latency is improved or not between instances on the same subnet. I did a little experiment and launched 20 instances in a single request. The private IPs I was assigned were scattered across three different subnets.<br /><br />We can run tests to determine the average network latency & speed across instances on the same and on different subnets. If these tests indicate that network is better within the same subnet then you can launch four or five times as many instances as you need, and terminate those that are not in the same subnet.<br /><br />Please share the results of any tests you do to determine the common subnet's effect on network latency.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-5069978078920877962009-12-25T11:20:39.943+02:002009-12-25T11:20:39.943+02:00Hello Shlomo,
This might be a unrelated questio...Hello Shlomo, <br /> This might be a unrelated question to this post, but since you are a EC2 expert, hope you can shed light on this.<br /><br /> I dont know is it only my observation or in general people have seen this.<br /><br />I was configuring some clusters on the EC2 and most importantly was putting up a typical mysql master-slave replication based cluster. As I was doing it, I saw that the every instance created was given a private IP from different subnets. When I did some basic network tests including ping round-trip test, I saw the network was fluctuating a lot and at time was miserable. This also reflected on the performance of the cluster, because they were made to communicate over private IPs.<br /><br />So wanted to know:<br />1. I guess there is no way you can ask for private IPs to be from the same subnet (except on VPC).<br />2. What I am seeing here, is it a normal phenomena on the Amazon?<br />3. If it is normal, how does one mitigate these issues. Cause with these basic network issues, performance at the backend is not at all good.<br /><br />Hoping that you will shed some light on this.Unknownhttps://www.blogger.com/profile/17503319820208932356noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-42856503384107869132009-12-25T02:47:02.191+02:002009-12-25T02:47:02.191+02:00@Matthew,
Thanks for pointing that out - I've...@Matthew,<br /><br />Thanks for pointing that out - I've edited it to say "at this point the client is communicating with one of your EC2 application instances". I hope that makes it clearer.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-50769853735359774432009-12-25T02:41:15.956+02:002009-12-25T02:41:15.956+02:00"4.The ELB virtual appliance at address 1.2.3..."4.The ELB virtual appliance at address 1.2.3.4 passes through the communications from the client to one of the EC2 instances in the load balancing pool. At this point the client is connected with one of your EC2 application instances"<br /><br />This is the part which I originally took to mean connections are handed off. But on reading it again after your reply, it makes a little more sense.<br /><br />The main thing is that it sounds like as long as I configure things correctly, the ELB will suit my requirements. Thanks again for the article!Unknownhttps://www.blogger.com/profile/02075758726320459682noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-73918192105439351972009-12-24T21:06:30.372+02:002009-12-24T21:06:30.372+02:00@Matthew,
I did not mean to imply that connection...@Matthew,<br /><br />I did not mean to imply that connections are "handed off" to back-end instances - they're not, the traffic is "passed through" to the back-end instances via the ELB. Please let me know where I might clarify the article.<br /><br />The theoretical bandwidth of ELB is unlimited - it will keep on scaling as long as the traffic keeps ramping up. So the overall bandwidth that your system will be able to handle is a direct result of the number of instances you put behind the ELB. As I described above, you'd need to test ELB carefully to make sure you're actually reproducing the conditions under which ELB was designed to scale. It's possible (and I've suggested as much in the forum) that the forum posts describing bandwidth limitations did not test the ELB properly and therefore hit a "faux" limit.<br /><br />Unlike ELB, software load balancers running on an EC2 instance (and hardware load balancers in a data center) cannot scale beyond the bandwidth of the network connection feeding into them.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-70600701274329420842009-12-24T20:41:01.908+02:002009-12-24T20:41:01.908+02:00Excellent blog, very informative.
I however, bein...Excellent blog, very informative.<br /><br />I however, being an EC2 newbie, would like to ask a quick question regarding bandwidth.<br /><br />Your explanation on how requests are handed off to instances (ie the client is eventually connected directly with an instance behind the ELB) suggests that max bandwidth should be increased when increasing the number of instances behing the ELB. Does this sound correct?<br /><br />What concerns me is that I've read several forum posts etc that seem to suggest this is not the case. And that bandwidth is limited to the bandwidth of the ELB itself and the number of instances behind is irrelevant. This doesn't sound right to me, hopefully they're wrong.<br /><br />Bandwidth is going to be very important for me, as my app will have a very heavy streaming element. So I am trying to determine whether or not I can increase bandwidth by simply increasing the number of instances I have behind my ELB.Unknownhttps://www.blogger.com/profile/02075758726320459682noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-7281510999597018882009-12-24T05:27:22.701+02:002009-12-24T05:27:22.701+02:00@Logan Green,
There's no API to modify the li...@Logan Green,<br /><br />There's no API to modify the listeners of an ELB. So you'll need to create a new ELB with both HTTP:80 and TCP:443 listeners specified at creation time.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-21468607917270151852009-12-24T05:17:51.625+02:002009-12-24T05:17:51.625+02:00Is it possible to edit a load balancer to forward ...Is it possible to edit a load balancer to forward additional ports? For example we launched our aws ELB forwarding port 80, and now we want to add https support and forward 443 as well. Do I need to create a new ELB or can I edit my existing one?Logan Greenhttps://www.blogger.com/profile/08127812758704904551noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-29111671102508675222009-12-10T15:35:52.496+02:002009-12-10T15:35:52.496+02:00Ah ha, thank you for solving the mystery. (And fo...Ah ha, thank you for solving the mystery. (And for introducing me to the word bi-coastal!)Max Christianhttps://www.blogger.com/profile/16945037488081634706noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-60802336472883548342009-12-06T03:41:58.572+02:002009-12-06T03:41:58.572+02:00@6p00e54ee6e7b68834,
That's also a good sugge...@6p00e54ee6e7b68834,<br /><br />That's also a good suggestion. Even better would be to use a single-use URL, which would cease to work after the first retrieval. Then it would not need to be deleted.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-61867268029370972702009-12-02T21:25:42.697+02:002009-12-02T21:25:42.697+02:00Extremely useful! This just gave me a great way to...Extremely useful! This just gave me a great way to correctly account for multiple clusters in StarCluster (http://web.mit.edu/starcluster)jtrileyhttps://www.blogger.com/profile/15026665697864370476noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-197285202865532962009-11-20T14:03:05.641+02:002009-11-20T14:03:05.641+02:00@Ramchandra,
1 GB/sec is not a lot and it is defi...@Ramchandra,<br /><br />1 GB/sec is not a lot and it is definitely possible to do with ELB. I've seen ELB handle more. I don't know what the limit is - if there is one, because there might not be a limit on the ELB side. The limit might be the number of back-end instances you can launch.<br /><br />There's no problem with putting all the back-end instances in the same availability zone.<br /><br />Concurrent connections are also not really limited by the ELB, but by the back-end instances.<br /><br />If your traffic patterns match those that ELB was designed for (i.e. gradually ramping-up) then it should be able to scale to handle that traffic.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-55749615522488685742009-11-20T05:04:58.394+02:002009-11-20T05:04:58.394+02:00hello Shlomo,
A very informative and well writte...hello Shlomo, <br /> A very informative and well written post, I am very thankful for that.<br /><br /> I just need to know you views on the following<br /><br />We are trying to achieve something of this order... 1 Million Requests per sec (1 KB size per request) == 1 gByte/sec. <br />* Can Amazon allow such a limit or in specific can it possible to achieve that with step 2 (ofcourse with proper backend on step4.)<br />* Can this be achieved with a single availability zone? <br />* Also we want to check on the concurrent connection limits. What is the limit of concurrency that Amazon supports? <br />* Can step 2 help us scale that as well.<br /><br />Hoping to hear your comments on this.. will surely be of much help to me.<br /><br />with regards<br /><br />SecionUnknownhttps://www.blogger.com/profile/17503319820208932356noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-73283379825418988002009-11-19T01:32:38.227+02:002009-11-19T01:32:38.227+02:00Comment from William Vambenepe, http://stage.vambe...Comment from William Vambenepe, http://stage.vambenepe.com/ :<br /><br />Before the contractor goes away, you should sit w/ them and go through the process of creating a new key pair and replacing the old one w/ the new one so you have him/her on hand if anything breaks (and if indeed that's the case, that the credential management in the system is a mess, then it's a huge red flag and you may want to question/verify everything the consultant did for you).<br /><br />Once they're gone you should change the credentials again, but at this point there is a low risk of it not working.<br /><br /><br />Shlomo says:<br /><br />Thanks for that excellent point, William. If the "simple" things don't work, why should you trust that the rest works as advertised.<br /><br />Sorry that this blog gave you trouble adding a comment - I needed to enable third-party cookies to allow me to authenticate as myself for comments (!).Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-59689238600097792222009-11-10T23:58:43.193+02:002009-11-10T23:58:43.193+02:00Given the constraints imposted by Auto Scaling, I ...Given the constraints imposted by Auto Scaling, I think there's a better option than using signed URLs.<br /><br />Signed URLs have the problem of a hardcoded expiration date, which means you need some external script which is vigilant in continually generating new signed URLs and updating your Auto Scaling Group parameters with the latest URL (which will need to be replaced again in X minutes). This puts robustness in direction opposition to security - the most secure solution mandates a short expiration time, which decreases the robustness of the system by requiring the external script to run frequently, without fail.<br /><br />There's a better solution that keeps all the security goodness of signed URLs with none of the "signed-URL-generator-must-run-or-Auto-Scaling-will-fail" badness.<br /><br />Instead of using signed URLs, use *public* URLs with a random path element:<br /><br />https://s3etc/as0df98a0b980a98a0sd98f0a98sdfa/secret-user-data.txt<br /><br />The URL is world-readable, but its path is unguessable (just like a signed URL).<br /><br />Your Auto Scaling Launch config is initially configured with this URL.<br /><br />Your external script then runs *whenever it wants*, creating a new random path & uploading your data to it, and then updating the Auto Scaling Launch Config to point at the new path. The script then deletes the file from the old path, so all running instances no longer have access to the secret data.<br /><br />This can be combined with the "wimpy" auth scheme so that the URL doesn't even need to be public, and thus your attacker (if lucky enough to remote-exec on the machine before the URL dies) needs more than just 'curl' to get the secret data.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-3872480357451508732009-10-27T15:50:29.755+02:002009-10-27T15:50:29.755+02:00Yeah, azure is kind of an odd duck right now. I&#...Yeah, azure is kind of an odd duck right now. I'm trying to find more about it but their web site is much harder to navigate and it's clear that they expect most of their users to access it via the very tightly integrated visual studio tools. Still, they do have REST interfaces behind a lot of these things which should make it possible to use them in different ways.Unknownhttps://www.blogger.com/profile/08349877802381481792noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-4938083950833617602009-10-27T15:22:59.367+02:002009-10-27T15:22:59.367+02:00@Mitch Garnaat,
True, I didn't include Racksp...@Mitch Garnaat,<br /><br />True, I didn't include Rackspace - they would go in the same box as Amazon and vCloud.<br /><br />I did struggle with where to put Azure. First, it's more of a Platform than Infrastructure, so I'm not sure it's 100% consistent to have it on the chart at all (since the rest of them are Infrastructure). Second, as you mention, it's both an API and a protocol. Third, it's not a standard at all.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-4967593053779603202009-10-27T14:30:35.336+02:002009-10-27T14:30:35.336+02:00Nice post. Couple of comments. You didn't in...Nice post. Couple of comments. You didn't include Rackspace, which is too bad. They have a REST API for their CloudServers product. Also, is it try that Azure is only an API? I think they have a REST API around management. They also have REST api's around their storage and queue services.Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-89046498482632645802009-10-27T12:22:41.773+02:002009-10-27T12:22:41.773+02:00Great info. Thanks for clarifying.Great info. Thanks for clarifying.About Ushttps://www.blogger.com/profile/14391621428560370602noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-42177603173161317062009-10-27T10:06:42.213+02:002009-10-27T10:06:42.213+02:00@Danny,
Um... thanks? :-)@Danny,<br /><br />Um... thanks? :-)Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-3275674360593784562009-10-27T04:44:32.810+02:002009-10-27T04:44:32.810+02:00I came to this expecting something awful. Read the...I came to this expecting something awful. Read the first couple of paragraphs, having seen plenty of monsters (check some of the Grid stuff) but - yes! You got it. (Soon I will read the rest of your piece :)Dannyhttps://www.blogger.com/profile/03703352768894605596noreply@blogger.com