Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [amazon-web-services]

Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.

4
votes
1answer
34 views

Running AWS commands works ok, but the code seems messy and I don't like the time.sleep bit

This is what I've come up with for a class that handles parsing AWS Textract output. So far it only gets the AWS Textract output into a .txt file. ...
3
votes
1answer
82 views

AWS Lambda function to update newly added DynamoDB records

Approach All data is submitted into my DynamoDB from another Lambda > API Integration function whereas the lastUpdated row gets inserted as ...
0
votes
0answers
30 views

Boto3 Python 2.7 class for encrypting Amazon images

I would appreciate a review of this Python code I have written, particularly with respect to general coding style, the design, and how I have organised the code as classes. Other feedback most welcome ...
1
vote
0answers
43 views

Jenkins Powershell script to copy code from Github to AWS Lambda

This is a <10 line Jenkins powershell build step triggered by a git commit which zips code and copies to lambda. Seems to work fine for javascript projects. I check for the zip file and then remove ...
3
votes
3answers
138 views

Upload .JSON product list to MongoDB and upload image to AWS S3

I'm using Node/Express/Mongoose to accept a JSON file containing a list of product details. These products are looped through, the images are uploaded to AWS S3, and the product is either accepted or ...
1
vote
0answers
203 views

Elastic Beanstalk configuration using ebextensions container_commands combining leader_only and test

For our Django app running on AWS Elastic Beanstalk, we use a web-server/worker setup. The same code is deployed to the web-server and worker environments, and environment-specific configuration is ...
4
votes
1answer
54 views

Simple server log backup script utilising AWS

I have server side log files written to various directories throughout the course of a week. I have the need to optionally compress, and archive these files to AWS. The script I have come up with ...
2
votes
1answer
37 views

More elegant way to construct the S3 path name with given options

I have a method that generates the S3 key prefix according to the provided args. However, my gut feeling is that this method is not as elegant as it should be, maybe I just don't have an eye for it. ...
6
votes
1answer
78 views

Select and delete objects from S3 container

GitLab saves files with the pattern 1530410429_2018_07_01_11.0.1_gitlab_backup.tar (epoch, date, version, fixed string) to an S3 container. I wanted to create a ...
2
votes
0answers
517 views

AWS Lambda Function to Register in Cognito

This is the entire code for a Lambda function that registers a new user in Amazon Cognito. The email, username, and password are passed to the Lambda function. The client ID is stored in an ...
0
votes
1answer
72 views

Validation of AWS tags

I have to validate Tags with Rules for AWS resource. Input contains list of dictionaries. ...
3
votes
3answers
648 views

Optimizing Node.js http get for AWS Lambda

I have the following Node.JS (runtime v6.10) executing on AWS Lambda: ...
2
votes
1answer
61 views

Creating a presigned URL

I've created software in Go to create a presigned URL: ...
1
vote
0answers
112 views

Go program to generate S3 presigned PUT URL

I'm have a made a simple Go program to generate a s3 presigned put url: ...
4
votes
0answers
495 views

AWS EC2 metadata fetcher in Python

Update https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html This seems to be the schema for the instance metadata, though the way they describe it isn't really handy for ...
2
votes
1answer
192 views

node.js connecting to s3 and youtube

I have a task to connect to S3 get a JSON file with the names of artists, use the youtube-data-api and search for the top 10 songs. Then send it back in a given ...
3
votes
1answer
33 views

S3 Bash Tools Part 3

This is a follow on to a previous question: Part 3: User Scripts All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify ...
1
vote
0answers
19 views

S3 Bash Tools Part 2

This is a follow on to a previous question: Part 2: Upload/Download All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly ...
1
vote
0answers
59 views

S3 Tools in Bash

This is a follow on to a previous question: Part 1: Signature All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify ...
1
vote
1answer
951 views

AWS Lambda Python function, to enable / disable access to a VPC environment

We have a requirement that access to an AWS environment is enabled on demand, and said access disabled afterwards. In each case (up / down), this means we will do two things: Enable access Turn on ...
4
votes
2answers
81 views

Bash Database Backup to AWS S3

I've written a bash script that backs up databases and uploads the backups to an S3 bucket. I have the bucket configured to delete backups that are older than 90 days. Is there anything that can be ...
3
votes
2answers
91 views

Authenticating to Amazon EC2 Container Registry

New to bash. The script I wrote works, but I don't believe it is written in the best way possible. The script only assesses how to execute the aws ecr get-login ...
4
votes
2answers
96 views

Sign an EC2/S3 URL Version 3

From the previous 2 versions: Sign an EC2/S3 URL Version 2 I fixed the comments from the previous version. Most of the code remains the same (so I have split out the new code for review). The main ...
2
votes
1answer
129 views

Checking a DB table before doing an insert using AWS Lambda

Pretty new to AWS Lambda and Javascript, and particularly Promises. Simple check db table before insert. It all seems to be working - I'm just wondering if there was a neater way of doing any of ...
2
votes
1answer
58 views

Sign an EC2/S3 URL Version 2

Follow up on previous review: Sign an EC2/S3 URL I have fixed the previous comments. I have also added the code needed to sign HTTP header fields as part of the request. ...
5
votes
2answers
148 views

Sign an EC2/S3 URL

The other day I found myself trying to walk through the instructions for signing an S3 URL. http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html Not as trivial as the ...
4
votes
1answer
823 views

Implement a boto3 RDS generator in Python

Boto3 does not seem to implement a generator for RDS instances, instead offering a marker and pagination feature. This code does work for my use case and I have been using it for the entire day ...
3
votes
1answer
336 views

Analyze terraform template files to return infrastructure cost

I am trying to write a program I am calling "terraform cashier." Terraform is a tool released by hashicorp to allow you to codify your infrastructure. I thought it would be useful to know how much ...
2
votes
0answers
155 views

Concurrent AWS S3 copying of images listed in MySQL

I created the following one off script and it did the task quite well (1000s of requests / second with low CPU usage - outperformed the previous python script in every aspect). But I am wondering if ...
4
votes
0answers
254 views

Test cases for Ansible lambda module for deployment of AWS Lambdas

I have written a set of test cases which cover the ansible lambda module. The entire source is available on GitHub, forked from ansible/ansible The original file is visible on github as is the ...
6
votes
1answer
2k views

Java 8 spliterator for paged results

Summary: I am using an API which returns paged results. I want to have these results as Java 8 Stream and implemented a ...
4
votes
0answers
506 views

Deploy a Python script as an AWS Lambda micro web service

I'm still new to python, but my history is in the C/C++ world with plenty of Makefiles, though I've rarely written from scratch such elaborate ones as this. The Hitchhiker's Guide to Python gave me ...
3
votes
0answers
155 views

ECMAScript 6 amazon lambda image processing

I use this script in amazon lambda for image processing. Script should do: download image and watermark from amazon S3 resize image to two sizes put watermark on it and annotate with text put resized ...
3
votes
1answer
57 views

Rewrite Amazon s3 key

I've create a function that rewrites the key or "path" of an object in s3. By default, Amazon Web Services Firehose writes to s3 in the format of ...
3
votes
1answer
1k views

Class of file uploading to S3 using boto with multipart supported

I'm writing an app by Flask with a feature to upload large file to S3 and made a class to handle this. I have some targets in writing code: Code must be easy to understand and maintain. (Yes, ...
1
vote
0answers
109 views

AWS Redshift wrapper class that automates similar types of loads from S3

I have a AWS Redshift wrapper class that automates similar types of loads from S3 for me, and I have recently adapted it to work for Spark jobs, which don't require a manifest, and instead need a ...
1
vote
1answer
1k views

Node.js put records to Kinesis with infinite retry strategy.

My application is really easy. It's going to add 500 records at a time to AWS Kinesis and if there's an error occur I'll just add that back to the queue and it's going to be retry. It's going well but ...
5
votes
2answers
111 views

Rails controller to manage EC2 instances

I did a small sample application for an Interview. Since I didn't got accepted I would like your opinion on how to improve it. The aux methods are private. I have a ...
4
votes
1answer
66 views

Simple shell script to backup to Amazon over iSCSI

The script works but I know that it can be improved. Please note that my scripting skills are very basic :) (as it can be seen in my Frankenstein script – I took bits and pieces of the code from ...
1
vote
0answers
519 views

Unit tests for a Redshift wrapper class

I'm somewhat new to TDD and unit testing, but I've written a suite of unit tests to check my functionality. The basic classes automate simple load and unload operations from s3 and redshift, and ...
2
votes
1answer
1k views

Redshift Load and Unload Wrapper class

We do constant traffic with our Redshift tables, and so I created a wrapper class that will allow for custom sql to be ran (or a default generic stmt), and can run a ...
2
votes
1answer
2k views

Listing EC2 instances

I've got this massive nested for loop with a few if and else statements in nodeJs. The output of all of this is just string which is combined from different values out of JSon object. What I want to ...
5
votes
1answer
5k views

Convert zip to gzip and upload to S3 bucket

My code accesses an FTP server, downloads a .zip file, pushes the file contents as .gz to an AWS S3 bucket. ...
5
votes
1answer
5k views

Testing boto connection

I am writing a unit test to be sure the code can connect to aws via boto, since I will be putting the credentials in place via some other process than code deployment. So far, I've only figured out ...
5
votes
1answer
485 views

Bash script to backup from S3 to GC

I'm working on a script to backup S3 to GC everyday. I'm not sure if my script has any potential bug or error that might have destroyed anything from both sides? The data is quite sensitive and I don'...
3
votes
1answer
1k views

Generating URL and status for an S3 bucket

The following program generates the S3 URL and response (status code?) for the key, provided the bucket name and if the S3 URL is a mess (wrong url), then it generates empty string as URL and error as ...
5
votes
3answers
70 views

Structuring code that shells out to an external script

This code that shells out to an external script could use some input in order to read smoother, I think. Praise and critique are welcome. ...
1
vote
1answer
3k views

Gets AWS VPC subnet info

I was hoping to get a quick review of a script I wrote for retrieving AWS VPC Subnet information. Any and all ideas welcome. I am looking specifically for logic enhancements, and optimizations to ...
4
votes
1answer
856 views

Uploading a video in S3 Using Future

I've posted a similar program previously, but I have not made any major modifications in the code, so I am posting it again by deleting the previous question. I am afraid of the thread keyword and I ...