A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [amazon-web-services]

Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.

Filter by
Sorted by
Tagged with
19
votes
3answers
76k views

List objects in a Amazon S3 folder without also listing objects in sub folders

I'm using the Amazon S3 Java SDK to fetch a list of files in a (simulated) sub-folder. This code is rather standard (AWSConfiguration is a class that contains a ...
8
votes
1answer
1k views

Connecting to EC2 instances using the instance name instead of IP

I'm an intensive user of AWS EC2 instances, many instances are launched, stopped, repurposed, etc. To connect to any instance using SSH I must keep track of their ...
6
votes
1answer
7k views

Testing boto connection

I am writing a unit test to be sure the code can connect to aws via boto, since I will be putting the credentials in place via some other process than code deployment. So far, I've only figured out ...
6
votes
1answer
119 views

Select and delete objects from S3 container

GitLab saves files with the pattern 1530410429_2018_07_01_11.0.1_gitlab_backup.tar (epoch, date, version, fixed string) to an S3 container. I wanted to create a ...
6
votes
1answer
2k views

Java 8 spliterator for paged results

Summary: I am using an API which returns paged results. I want to have these results as Java 8 Stream and implemented a ...
5
votes
3answers
70 views

Structuring code that shells out to an external script

This code that shells out to an external script could use some input in order to read smoother, I think. Praise and critique are welcome. ...
5
votes
1answer
500 views

Bash script to backup from S3 to GC

I'm working on a script to backup S3 to GC everyday. I'm not sure if my script has any potential bug or error that might have destroyed anything from both sides? The data is quite sensitive and I don'...
5
votes
2answers
115 views

Rails controller to manage EC2 instances

I did a small sample application for an Interview. Since I didn't got accepted I would like your opinion on how to improve it. The aux methods are private. I have a ...
5
votes
1answer
6k views

Convert zip to gzip and upload to S3 bucket

My code accesses an FTP server, downloads a .zip file, pushes the file contents as .gz to an AWS S3 bucket. ...
4
votes
2answers
85 views

Bash Database Backup to AWS S3

I've written a bash script that backs up databases and uploads the backups to an S3 bucket. I have the bucket configured to delete backups that are older than 90 days. Is there anything that can be ...
4
votes
1answer
952 views

Implement a boto3 RDS generator in Python

Boto3 does not seem to implement a generator for RDS instances, instead offering a marker and pagination feature. This code does work for my use case and I have been using it for the entire day ...
4
votes
1answer
872 views

Uploading a video in S3 Using Future

I've posted a similar program previously, but I have not made any major modifications in the code, so I am posting it again by deleting the previous question. I am afraid of the thread keyword and I ...
4
votes
2answers
105 views

Sign an EC2/S3 URL Version 3

From the previous 2 versions: Sign an EC2/S3 URL Version 2 I fixed the comments from the previous version. Most of the code remains the same (so I have split out the new code for review). The main ...
4
votes
2answers
178 views

Sign an EC2/S3 URL

The other day I found myself trying to walk through the instructions for signing an S3 URL. http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html Not as trivial as the ...
4
votes
1answer
235 views

Running AWS commands works ok, but the code seems messy and I don't like the time.sleep bit

This is what I've come up with for a class that handles parsing AWS Textract output. So far it only gets the AWS Textract output into a .txt file. ...
4
votes
1answer
72 views

Simple server log backup script utilising AWS

I have server side log files written to various directories throughout the course of a week. I have the need to optionally compress, and archive these files to AWS. The script I have come up with ...
4
votes
1answer
70 views

Simple shell script to backup to Amazon over iSCSI

The script works but I know that it can be improved. Please note that my scripting skills are very basic :) (as it can be seen in my Frankenstein script – I took bits and pieces of the code from ...
4
votes
0answers
848 views

AWS EC2 metadata fetcher in Python

Update https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html This seems to be the schema for the instance metadata, though the way they describe it isn't really handy for ...
4
votes
0answers
299 views

Test cases for Ansible lambda module for deployment of AWS Lambdas

I have written a set of test cases which cover the ansible lambda module. The entire source is available on GitHub, forked from ansible/ansible The original file is visible on github as is the ...
4
votes
0answers
547 views

Deploy a Python script as an AWS Lambda micro web service

I'm still new to python, but my history is in the C/C++ world with plenty of Makefiles, though I've rarely written from scratch such elaborate ones as this. The Hitchhiker's Guide to Python gave me ...
3
votes
3answers
150 views

Upload .JSON product list to MongoDB and upload image to AWS S3

I'm using Node/Express/Mongoose to accept a JSON file containing a list of product details. These products are looped through, the images are uploaded to AWS S3, and the product is either accepted or ...
3
votes
2answers
132 views

Authenticating to Amazon EC2 Container Registry

New to bash. The script I wrote works, but I don't believe it is written in the best way possible. The script only assesses how to execute the aws ecr get-login ...
3
votes
1answer
59 views

Rewrite Amazon s3 key

I've create a function that rewrites the key or "path" of an object in s3. By default, Amazon Web Services Firehose writes to s3 in the format of ...
3
votes
1answer
100 views

AWS Lambda function to update newly added DynamoDB records

Approach All data is submitted into my DynamoDB from another Lambda > API Integration function whereas the lastUpdated row gets inserted as ...
3
votes
3answers
694 views

Optimizing Node.js http get for AWS Lambda

I have the following Node.JS (runtime v6.10) executing on AWS Lambda: ...
3
votes
1answer
353 views

Analyze terraform template files to return infrastructure cost

I am trying to write a program I am calling "terraform cashier." Terraform is a tool released by hashicorp to allow you to codify your infrastructure. I thought it would be useful to know how much ...
3
votes
1answer
1k views

Generating URL and status for an S3 bucket

The following program generates the S3 URL and response (status code?) for the key, provided the bucket name and if the S3 URL is a mess (wrong url), then it generates empty string as URL and error as ...
3
votes
0answers
36 views

SageMakerTrainingOperator in Airflow

I have made an operator (surrounded by others operators) for training a model in sagemaker in airflow and I have doubts how would it be more readable or more pythonic. First I did this: ...
3
votes
0answers
81 views

User registration in Actix w/ DynamoDB

I am trying to puzzle out api based user registration for a service I am building, but I have a strong feeling that my code is not optimal. It feels like passing "state" in the way I did prevents the ...
3
votes
0answers
737 views

AWS Lambda Function to Register in Cognito

This is the entire code for a Lambda function that registers a new user in Amazon Cognito. The email, username, and password are passed to the Lambda function. The client ID is stored in an ...
3
votes
0answers
166 views

ECMAScript 6 amazon lambda image processing

I use this script in amazon lambda for image processing. Script should do: download image and watermark from amazon S3 resize image to two sizes put watermark on it and annotate with text put resized ...
2
votes
1answer
2k views
2
votes
1answer
4k views

Python Boto Script - Sorting based on date

I had a question regarding my code which downloads a file from S3 with the highest (most recent) timedated filename format: YYYYMMDDHHMMSS.zip ...
2
votes
1answer
39 views

More elegant way to construct the S3 path name with given options

I have a method that generates the S3 key prefix according to the provided args. However, my gut feeling is that this method is not as elegant as it should be, maybe I just don't have an eye for it. ...
2
votes
1answer
34 views

S3 Bash Tools Part 3

This is a follow on to a previous question: Part 3: User Scripts All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify ...
2
votes
1answer
682 views

Amazon S3 pass-through controller

I am building a website that uses Amazon S3 to host its image (and other) files. I require that images displayed on the website look like they come from my site and not from Amazon (so we are talking ...
2
votes
1answer
91 views

Creating a presigned URL

I've created software in Go to create a presigned URL: ...
2
votes
1answer
292 views

node.js connecting to s3 and youtube

I have a task to connect to S3 get a JSON file with the names of artists, use the youtube-data-api and search for the top 10 songs. Then send it back in a given ...
2
votes
1answer
220 views

Checking a DB table before doing an insert using AWS Lambda

Pretty new to AWS Lambda and Javascript, and particularly Promises. Simple check db table before insert. It all seems to be working - I'm just wondering if there was a neater way of doing any of ...
2
votes
1answer
60 views

Sign an EC2/S3 URL Version 2

Follow up on previous review: Sign an EC2/S3 URL I have fixed the previous comments. I have also added the code needed to sign HTTP header fields as part of the request. ...
2
votes
1answer
2k views

Class of file uploading to S3 using boto with multipart supported

I'm writing an app by Flask with a feature to upload large file to S3 and made a class to handle this. I have some targets in writing code: Code must be easy to understand and maintain. (Yes, ...
2
votes
1answer
1k views

Redshift Load and Unload Wrapper class

We do constant traffic with our Redshift tables, and so I created a wrapper class that will allow for custom sql to be ran (or a default generic stmt), and can run a ...
2
votes
1answer
2k views

Listing EC2 instances

I've got this massive nested for loop with a few if and else statements in nodeJs. The output of all of this is just string which is combined from different values out of JSon object. What I want to ...
2
votes
1answer
90 views

Exporting data from PostgreSQL as CSV to S3 bucket

I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
2
votes
0answers
46 views

Validating Amazon CloudFormation templates using Rspec

I have created an open source project that uses Rspec to validate the formatting of AWS::CloudFormation::Init YAML blocks in CloudFormation templates.1 My project is here. Some general concerns: Is ...
2
votes
0answers
681 views

Elastic Beanstalk configuration using ebextensions container_commands combining leader_only and test

For our Django app running on AWS Elastic Beanstalk, we use a web-server/worker setup. The same code is deployed to the web-server and worker environments, and environment-specific configuration is ...
2
votes
0answers
161 views

Concurrent AWS S3 copying of images listed in MySQL

I created the following one off script and it did the task quite well (1000s of requests / second with low CPU usage - outperformed the previous python script in every aspect). But I am wondering if ...
1
vote
1answer
89 views

Python functions to save JSON and H5 to local files and S3

I'm trying to copy files to S3 after writing them locally: I have 2 functions which I need to do this. What is the best way to refactor the code that exists in both. ...
1
vote
1answer
4k views

Gets AWS VPC subnet info

I was hoping to get a quick review of a script I wrote for retrieving AWS VPC Subnet information. Any and all ideas welcome. I am looking specifically for logic enhancements, and optimizations to ...
1
vote
1answer
33 views

Enumerating AWS instances in Rails, with error handling

I have created a method, which is raising the same exception in case of any error: ...