2022 Developer Survey is open! Take survey.

Questions tagged [awk]

AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool.

Filter by
Sorted by
Tagged with
4 votes
1 answer
48 views

Awk program for extracting unique values from a k1=v1,k2=v2,... list

A single string consisting of comma-separated pairs in the format <key>=<value>, like ...
user avatar
1 vote
1 answer
34 views

Extract sections of a file into separate files

I have a file of the form - >SDF123.1 blah blah ATCTCTGGAAACTCGGTGAAAGAGAGTAT AGTGATGAGGATGAGTGAG... >SBF123.1 blah blah ATCTCTGGAAACTCGGTGAAAGAGAGTAT AGTGATGAGGATGAGTGAG.... And I want to ...
user avatar
  • 135
0 votes
0 answers
33 views

awk script to convert simple markup language to troff or html

I created a simple Unicode-based markup language called Incipit and an awk(1) script to convert a text written in Incipit into html or troff -ms. A example of text ...
user avatar
  • 854
0 votes
0 answers
36 views

UNIX calendar(1) in awk

This is a simple implementation of the calendar(1) utility included in some UNIX systems (all BSDs have it, GNU has not). Here's the manual: ...
user avatar
  • 854
1 vote
0 answers
41 views

detection the selfish node Using awk File [closed]

I am working on detection the selfish node based on this equation Using awk File : (Node Communication Ratio) NCR= ((Get Route Request-(Get Route Request-Reply Route Request))/Get Route Request)*100 ...
user avatar
  • 19
2 votes
0 answers
50 views

The Blacklist v2

This script gathers domains listed from sources in this JSON file and compiles them into one text file. Text files that have a similar key under the whitelists ...
user avatar
  • 2,897
3 votes
1 answer
65 views

Parse log file and send the result to an API

The task is to get the most recent login time of the day for each user. Then send such a result to an API. The log file (file.log) looks like the following (only ...
user avatar
  • 5,487
2 votes
0 answers
33 views

bash / awk workflow operating on multi-column files [closed]

I am working on the improvement of the bash-awk workflow designed for the analysis of data in multiple-colums format. The script uses AWK code (which extract numbers from selected columns of the input....
user avatar
  • 139
1 vote
0 answers
25 views

Printing of status message during execution of the bash script

As a part of my bash routine I am printing some message in terminal regarding status of the workflow. The message in splited into two parts (part 1: begining of the task, part 2: status of its ...
user avatar
  • 139
0 votes
1 answer
92 views

Compare script using awk and bash

I try to create compare script using awk and another command and running successful. But I think the script I create it's to long. Is there anyone can shorten my script below ? After plan A ...
user avatar
  • 11
2 votes
0 answers
46 views

Faster algorithm/language for substring sliding window replacement

I had a sample file with a long string >50M like this. ...
user avatar
  • 21
2 votes
1 answer
127 views

Bash script wallpaper randomizer

I thought it would be fun to write a simple and small bash script which randomly chooses a wallpaper from a given directory of a set of wallpaper images. The code is as follows. ...
user avatar
1 vote
1 answer
72 views

Parse and clean large log files

I have the code below, which works successfully, and is used to parse, clean log files (very large in size) and output into smaller sized files. Output filename is the first 2 characters of each line. ...
user avatar
0 votes
1 answer
26 views

Split large log file like a dictionary [closed]

I have a large log file in the following format. Do note that this is already sorted alphabetically. I wish to split it into smaller pieces, and with file names reflecting the content of the file (...
user avatar
1 vote
1 answer
62 views

Extracting emails from log files

I have huge number of text files, each several hundred MB in size. Unfortunately, they are not all fully standardized in any one format. Plus there is a lot of legacy in here, and a lot of junk and ...
user avatar
3 votes
1 answer
316 views

An Awk program to find all primes till N

I developed Awk code to print all primes till N where N is given as command line argument. I think I took pretty much all the optimizations into account. Could someone help if it could be optimized ...
user avatar
3 votes
0 answers
36 views

Merging multiple awk commands while retaining functionality

Turned in a project a few days ago, and I needed a way to filter and format values off a file. I decided to go for a bash script since it seemed like a good option, and I came up with the following: <...
user avatar
  • 31
3 votes
1 answer
798 views

Extract multiple attributes values in XML tags

I have a XML file (test.xml) that can be sum up as follow (I filtered it so it will be more readable): ...
user avatar
  • 287
2 votes
0 answers
83 views

Bash wrapper for connmanctl with autocompletion

So the idea came from the fact that connmanctl only completes filenames/directories when used non-interactively, whereas when used interactively (executing ...
user avatar
  • 497
6 votes
3 answers
420 views

Count unique IP address in a date range from Apache log

I use the following command to get specific entries between certain dates from Apache log files. Then I filter out my own address and some bots, then I print out the IP, sort it, get the uniques, ...
user avatar
4 votes
1 answer
79 views

Removing duplicate field entries from sorted csv data

Given the following input (cat i.txt), I want to remove duplicate field entries in each of the first three columns and none of the others. ...
user avatar
0 votes
2 answers
76 views

Select a specific line after matching a pattern

I just recently started using AWK and I'm still learning about it. I have solved the problem I'm about to show but I feel it's not the best solution and I'm trying to find a solution to fit within an ...
user avatar
  • 3
4 votes
1 answer
83 views

Select second field in text table with shell script

I am writing a bash script to mount openshift service accounts into kubernetes objects. The right secret to use is highlighted in this text: $ oc describe sa sa-...
user avatar
  • 308
3 votes
2 answers
99 views

Code to get the IP address and format it to run in nmap

For scanning my subnet using my IP address, I need to run the following command nmap -sn 192.168.100.* I want to extend this functionality such that I don't have ...
user avatar
4 votes
0 answers
50 views

Summary data using CSV file

Is there the possibility to improve the code below? I am sure the result I got can be achieved with shorter code. Code ...
user avatar
  • 173
7 votes
1 answer
1k views

Updating my package version using only a Makefile

So I have written a Makefile that has a release function that can be run like: make release bump=patch and I want it to be fully ...
user avatar
  • 243
7 votes
3 answers
640 views

Print code-fenced sections of a Markdown document

Original code and demo at this gist. Given a Markdown document like ...
user avatar
  • 1,159
3 votes
1 answer
52 views

Counts and average code

Using the columns 4 and 2 will create a report like the output file showed below. My code works fine but I believe it can be done shorter. I have a doubt in the part of the split: ...
user avatar
  • 173
3 votes
2 answers
171 views

Bash script for managing hashtag notes

I have written a simple script for managing hashtag notes. I am not familiar with bash scripting so I would really appreciate it if someone could give me some pointers about it. ...
user avatar
  • 33
4 votes
1 answer
49 views

Remove every line that include and follows the second occurrence of a given pattern

From a file, I am trying to remove every line that include and follows the second occurrence of a given pattern in Bash (Mac OSX). Note, that a file may have only one match of the given pattern (and ...
user avatar
  • 607
4 votes
1 answer
86 views

Try every option for a command and make anagrams out of the successful ones

What I have: I am trying to find what words you can make out of Linux command options like ls -cat or ps -elf. To do that I am ...
user avatar
2 votes
1 answer
151 views

Extract thoughput value from JMeter log file

I use JMeter for load testing and want to report error on CI pipeline when application though put is less than 1/sec. JMeter logs summary of results as - ...
user avatar
  • 165
3 votes
1 answer
103 views

ECG Bash selection tool

I made the following bash script for extracting a group of ECG signals from ECG files. I would like to know if there is any mistakes and/or weaknesses. I have experienced difficulties in integrating ...
user avatar
5 votes
1 answer
3k views

Python library for awk-like file manipulation

I recently published a library for advanced awk-like file manipulation in Python 3. The code can be found here and here is the documentation. It is also available for download from pip (...
user avatar
3 votes
1 answer
370 views

Parsing GTF file using command-line

I am extracting exons details from a GTF file using command line in Unix like cut, awk, grep or sed. input file.gtf: ...
user avatar
  • 131
4 votes
2 answers
406 views

Generate SQL UPDATE from Excel CSV file

I must write updates in a database from a flat file (CSV). I want to do that in the shell, with tools such as AWK. ...
user avatar
4 votes
1 answer
204 views

Find and report all occurrences of duplicate lines in a text file

I would like you to take a look at this simple script I wrote that's supposed to emulate the Unix sort [FILE] | uniq -cd command. What's different about my script ...
user avatar
  • 297
0 votes
1 answer
38 views

Loading a file in vim via piped output of ag

I feel as if my scenario is fairly simple but that I might be overcomplicating it. I am searching through files looking for a special condition, that is where the method ...
user avatar
  • 439
2 votes
1 answer
684 views

Finding hidden field values in a web page before submitting a login form

In order to submit a login form using wget, I need to find the values for the hidden field that were added to the form recently. Grepping for the names gives the ...
user avatar
2 votes
1 answer
60 views

Listing Docker Images without registry URI

I have a private container registry behind a firewall that will need some Google registry images. To make sure things are updated sanely, I'm writing a mirror script in Bash. I have one line that ...
user avatar
  • 463
3 votes
2 answers
241 views

Advanced CSV-to-SQLite converter

I just wrote an advanced CSV parser that translates CSV to SQL(SQLite Compatible) statements, which are then inserted into the database. It translates variable names in the CSV to values defined in ...
user avatar
  • 875
1 vote
1 answer
34 views

Print first element of the last line of files when the second element does not match a pattern

I am relatively new to Bash, regular expression and stuff. In a directory, I need to read the last line of all the files that contain the word "fitness", and list (on one line and separated with ...
user avatar
  • 607
6 votes
3 answers
256 views

Shell script to count chess game outcomes

I came across this blog post by Adam Drake from around a year ago which is now making the rounds again. I made some improvements to his code, but wish to see if there are additional tweaks that could ...
user avatar
  • 351
7 votes
1 answer
76 views

Calculating percentages in arbitrary number of columns

Given this sample input: ...
user avatar
  • 107k
5 votes
2 answers
3k views

Checking percentage of free memory using top and awk

The following Awk code was created in order to verify if free memory from the top command is less than 20% of the total. I've noticed that I actually do not need the ...
user avatar
3 votes
1 answer
2k views

Analyse an Nginx access log with awk and produce a report

I'm looking for your opinions on anything that I am doing wrong in the application below, such as best practices, glaringly horrible errors or even just your own personal opinion. Task: Grab an Nginx ...
user avatar
10 votes
3 answers
25k views

Extracting data from text file in bash using awk, grep, head and tail

I've been writing bash script on and off, with pretty good results in terms of getting the job done. However, I'm worried that my script might be very ugly, as I am a beginner. I'm looking for advice ...
user avatar
  • 103
4 votes
2 answers
108 views

Filter shell script to find lines that contain all specified patterns

I wrote a script that does the following: Run another script on the system Filter the output to find lines that contain ALL of the given patterns Pipe the output to a second script on the system I ...
user avatar
  • 938
4 votes
1 answer
75 views

Extracting performance statistics from FTP session transcript

I've borrowed and written the following code to output the disconnect time. All works well but I'm curious as to how I could tighten/ shorten the code. If anyone feels like having some fun then I'd ...
user avatar
  • 41
9 votes
1 answer
984 views

Bash script - automate an upload image + description process on a server

I was tasked at work to improve a system where through a web interface a client, which eventually became us, uploads a bunch of images with descriptions (latter coming from csv files), the images are ...
user avatar