2
\$\begingroup\$

All help welcomed!

I posted a big chunk of code here - but feel free to review only a small section. I will use answers to improve this deployed open source software.

About the code

This code works. But can be improved in code quality. Feedback is highly welcomed!

codespell, ktlint, Android Studio validator were already used and followed where it made sense.

Format

If you make a nontrivial suggestions: please add note that you license your work on GPLv3 license. It is not needed for trivial changes, but for substantial I need this to be able to use your work.

I will mention your SO nick in the commit message and link your answer as thanks. Please specify preferred name and email if you want for commit credit. Or request anonymous contribution.

Explaining the program

Background

StreetComplete is an editor of OpenStreetMap database.

OpenStreetMap is an openly licenced geographic database where objects are represented by

  1. geometries - lines/ways/areas/etc.

  2. key-value pairs called tags describing kind of object. For example

Typical StreetComplete edits is adding surface=asphalt or surface=dirt to way representing road - to mark road surface.

Finally getting close

Taginfo project listing is used for publishing a simple .json file listing tags used by a given project.

So, this code presented here will parse source code files of StreetComplete to detect which tags can be added/edited, and will list it in a .json file.

Example, everyone loves examples

For example here is defined quest asking whether public toilet is paid.

Among other parts it has:

    override fun applyAnswerTo(answer: Boolean, tags: Tags, timestampEdited: Long) {
        tags["fee"] = answer.toYesNo()
    }

in this case key fee can be set to either fee=yes or fee=no as toYesNo() cannot return anything else.

There are over 100 other quests, some significantly more complex.

Design decisions

In several places shortcuts were taken to avoid very complicated parsing.

As this is tightly coupled to StreetComplete, many assumptions can be made.

Some places are marked with TODOs and I am not expecting help with them.

Running the code

To run code:

git clone https://github.com/matkoniecz/StreetComplete.git
cd StreetComplete
git checkout code-review
./gradlew updateTaginfoListing

which should work (may require installing gradle/kotlin)

Actual code

If someone is interested in looking at full deployed code - see https://github.com/matkoniecz/Zazolc/blob/taginfo/buildSrc/src/main/java/UpdateTaginfoListingTask.kt (on taginfo branch)

To keep it within character and sanity limits I stripped part of the less relevant code, especially part which hardcoded answers.

It is still fully functional.

import com.charleskorn.kaml.Yaml
import com.charleskorn.kaml.YamlConfiguration
import kotlinx.ast.common.AstSource
import kotlinx.ast.common.ast.Ast
import kotlinx.ast.common.ast.AstNode
import kotlinx.ast.common.ast.DefaultAstNode
import kotlinx.ast.common.ast.DefaultAstTerminal
import kotlinx.ast.common.ast.astInfoOrNull
import kotlinx.ast.common.klass.KlassDeclaration
import kotlinx.ast.common.klass.KlassIdentifier
import kotlinx.ast.common.klass.KlassString
import kotlinx.ast.common.klass.StringComponentRaw
import kotlinx.ast.grammar.kotlin.common.summary
import kotlinx.ast.grammar.kotlin.target.antlr.kotlin.KotlinGrammarAntlrKotlinParser
import kotlinx.serialization.ExperimentalSerializationApi
import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromString
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
import org.gradle.api.DefaultTask
import org.gradle.api.tasks.Input
import org.gradle.api.tasks.TaskAction
import java.io.File
import java.io.InputStream
import kotlin.system.exitProcess

/* Generate Taginfo tag listing - only tags added or edited are listed
 * Tags removed or used in filtering are NOT listed.
 *
 * Follows https://wiki.openstreetmap.org/wiki/Taginfo/Projects documentation
 */

@OptIn(ExperimentalSerializationApi::class) // needed by explicitNulls = false
open class UpdateTaginfoListingTask : DefaultTask() {

    @get:Input var targetDir: String? = null

    companion object {
        const val NAME_OF_FUNCTION_EDITING_TAGS = "applyAnswerTo"
        const val KOTLIN_IMPORT_ROOT_WITH_SLASH_ENDING = "app/src/main/java/"
        const val QUEST_ROOT_WITH_SLASH_ENDING = "app/src/main/java/de/westnordost/streetcomplete/quests/"
        const val COUNTRY_METADATA_PATH_WITH_SLASH_ENDING = "app/src/main/assets/country_metadata/"
        // is it possible to use directly SC constants?
        // import de.westnordost.streetcomplete.osm.SURVEY_MARK_KEY
        const val SURVEY_MARK_KEY = "check_date"
        const val VIBRATING_BUTTON = "traffic_signals:vibration"
        private const val SOUND_SIGNALS = "traffic_signals:sound"
    }

    private fun generateReport(questData: List<TagQuestInfo>) {
        println(targetDir)
        val format = Json { encodeDefaults = true; explicitNulls = false; prettyPrint = true  }

        @Serializable
        data class TagWithDescriptionForTaginfoListing(val key: String, val value: String?, val description: String)

        @Serializable
        data class Project(val name: String, val description: String, val project_url: String, val doc_url: String, val icon_url: String, val contact_name: String, val contact_email: String)

        @Serializable
        data class TaginfoReport(val data_format: Int = 1, val data_url: String, val project: Project, val tags: List<TagWithDescriptionForTaginfoListing>)

        fun showChangesMadeIfAny(report: TaginfoReport, oldReport: TaginfoReport) {
            if (report.tags != oldReport.tags) {
                println("new tags are different! verify that")
                report.tags.forEach {
                    if (it !in oldReport.tags) {
                        println("new entry: $it")
                    }
                }
                oldReport.tags.forEach {
                    if (it !in report.tags) {
                        println("removed entry: $it")
                    }
                }
            }
        }

        // https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/json.md
        val project = Project("StreetComplete", "Surveyor app for Android",
            "https://github.com/westnordost/StreetComplete",
            "https://wiki.openstreetmap.org/wiki/StreetComplete",
            "https://raw.githubusercontent.com/westnordost/StreetComplete/master/app/src/main/res/mipmap-xhdpi/ic_launcher.png",
            "my name, snipped in SO",
            "my email, snipped in SO",
        )

        val dataUrl = "snipped in SO due to character limit"
        val report = TaginfoReport(1, dataUrl, project,
            questData.map { TagWithDescriptionForTaginfoListing(it.tag.key, it.tag.value, "added or edited tag") }
            )
        val jsonText = format.encodeToString(report)
        val targetFile = File(targetDir, "taginfo_listing_of_tags_added_or_edited_by_StreetComplete.json")
        if (targetFile.exists()) {
            val oldText = targetFile.readText()
            val oldReport = format.decodeFromString<TaginfoReport>(oldText)
            showChangesMadeIfAny(report, oldReport)
        }
        val fileWriter = targetFile.writer()
        fileWriter.write(jsonText)
        fileWriter.close()
    }

    @TaskAction fun run() {
        println(targetDir)

        val foundTags = mutableListOf<TagQuestInfo>()
        val folderGenerator = questFolderGenerator()

        while (folderGenerator.hasNext()) {
            val folder = folderGenerator.next()

            File(folder.toString()).walkTopDown().forEach {
                if (it.isFile) {
                    if (it.name in listOf("AddAddressStreet.kt", "AddRoadName.kt", "AddStreetParking.kt",
                            "AddMaxSpeed.kt", "AddSidewalk.kt", "AddWayLit.kt", "AddMaxWeight.kt",
                            "AddStepsRamp.kt", "AddDrinkingWater.kt", "AddRecyclingContainerMaterials.kt", "AddBuildingType.kt",
                            "AddRoadSurface.kt", "AddPathSurface.kt", "AddFootwayPartSurface.kt", "AddCyclewayPartSurface.kt",
                            "AddStileType.kt", "AddCyclewayWidth.kt", "AddCycleway.kt", "AddSidewalkSurface.kt",
                            "AddPitchSurface.kt", "AddBikeParkingFee.kt", "AddParkingFee.kt",
                            "AddOrchardProduce.kt")) {
                        println("skipping $it")
                    } else if (isQuestFile(it)) {
                        addedOrEditedTags(it)!!.forEach { tags ->
                            foundTags.add(TagQuestInfo(tags, it.name))
                        }
                    }
                }
            }
        }
        generateReport(foundTags)
        reportResultOfDataCollection(foundTags)
    }

    private fun questFolderGenerator() = iterator {
        File(QUEST_ROOT_WITH_SLASH_ENDING).walkTopDown().maxDepth(1).forEach { folder ->
            if (folder.isDirectory && "$folder/" != QUEST_ROOT_WITH_SLASH_ENDING) {
                yield(folder)
            }
        }
    }

    private fun isQuestFile(file: File): Boolean {
        if (".kt" !in file.name) {
            return false
        }
        listOf("Form.kt", "Adapter.kt", "Utils.kt").forEach { if (it in file.name) {
                return false
            }
        }
        if (file.name == "AddressStreetAnswer.kt") {
            return false
        }
        if ("Add" in file.name || "Check" in file.name || "Determine" in file.name || "MarkCompleted" in file.name) {
            return true
        }
        return false
    }

    private fun reportResultOfDataCollection(foundTags: MutableList<TagQuestInfo>) {
        println("${foundTags.size} entries registered")
        val tagsFoundPreviously = 1619
        if (foundTags.size != tagsFoundPreviously) {
            println("Something changed in processing! foundTags count ${foundTags.size} vs $tagsFoundPreviously previously")
        }
    }

    private fun streetCompleteIsReusingAnyValueProvidedByExistingTagging(questDescription: String, key: String): Boolean {
        // much too complicated and error prone and rare to get that info by parsing
        if ("MarkCompletedHighwayConstruction" in questDescription && key == "highway") {
            return true
        }
        if ("MarkCompletedBuildingConstruction" in questDescription && key == "building") {
            return true
        }
        return false
    }

    private fun freeformKey(key: String): Boolean {
        // most have own syntax and limitations obeyed by SC
        // maybe move to general StreetComplete file about OSM tagging?
        if (key in listOf("name", "int_name", "ref",
                "addr:flats", "addr:housenumber", "addr:street", "addr:place", "addr:block_number", "addr:streetnumber",
                "addr:conscriptionnumber", "addr:housename",
                "building:levels", "roof:levels", "level",
                "collection_times", "opening_hours", "opening_date", "check_date",
                "fire_hydrant:diameter", "maxheight", "width", "cycleway:width",
                "maxspeed", "maxspeed:advisory", "maxstay",
                "maxweight", "maxweightrating", "maxaxleload", "maxbogieweight",
                "maxspeed:type", // not really true, but I give up for now. TODO: remove
                "capacity", "step_count",
                "lanes", "lanes:forward", "lanes:backward", "lanes:both_ways",
                "turn:lanes:both_ways", "turn:lanes", "turn:lanes:forward", "turn:lanes:backward",
                "operator", // technically not fully, but does not make sense to list all that autocomplete values
                "brand",
                "sport", // sport=soccer;volleyball is fully valid - doe not entirely fit here but...
                "produce", // like sport=*
            )) {
            return true
        }
        if (SURVEY_MARK_KEY in key) {
            return true
        }
        if (key.endsWith(":note")) {
            return true
        }
        if (key.endsWith(":conditional")) {
            return true
        }
        if (key.endsWith(":wikidata")) {
            return true
        }
        if (key.endsWith(":wikipedia")) {
            return true
        }
        if (key.startsWith("lanes:")) {
            return true
        }
        if (key.startsWith("name:")) {
            return true
        }
        if (key.startsWith("source:")) {
            return true
        }
        return false
    }

    private fun loadFileText(file: File): String {
        val inputStream: InputStream = file.inputStream()
        return inputStream.bufferedReader().use { it.readText() }
    }

    @Serializable
    class Tag(val key: String, val value: String?) {
        override fun toString(): String {
            if (value == null) {
                return "$key=*"
            }
            return "$key=$value"
        }

        override fun equals(other: Any?): Boolean {
            if (this === other) return true
            if (other !is Tag) return false
            if (key != other.key) return false
            if (value != other.value) return false
            return true
        }

        override fun hashCode(): Int {
            var result = key.hashCode()
            result = 31 * result + (value?.hashCode() ?: 0)
            return result
        }
    }

    class TagQuestInfo(val tag: Tag, val quest: String) {
        override fun toString(): String {
            return "$tag in $quest"
        }
    }

    private fun addedOrEditedTags(file: File): Set<Tag>? {
        val hardcodedAnswers = addedOrEditedTagsHardcodedAnswers(file)
        if (hardcodedAnswers != null) {
            return hardcodedAnswers
        }
        val suspectedAnswerEnumFiles = candidatesForEnumFilesForGivenFile(file)
        val description = file.parentFile.name + File.separator + file.name
        val fileSourceCode = loadFileText(file)
        return addedOrEditedTagsRealParsing(description, fileSourceCode, suspectedAnswerEnumFiles)
    }

    private fun addedOrEditedTagsHardcodedAnswers(file: File): Set<Tag>? {
        val fileSourceCode = loadFileText(file)
        val description = file.parentFile.name + File.separator + file.name
        var suspectedAnswerEnumFiles = candidatesForEnumFilesForGivenFile(file)
        // TODO hardcoding is ugly and ideally would be replaced
        // this function contains cases where answers are partially or fully hardcoded
        // it is done this way as in some cases parsing would extremely complex and not worth doing this
        // in some it can be actually implemented and it is likely worth doing this to avoid need
        // for manual maintenance of the code
        if ("AddBarrier" in file.name) { // outside when switch to try covering also unlikely new AddBarrier quests
            // TODO argh? can it be avoided?
            // why it is present? Without this AddBarrierOnPath would pull also StileTypeAnswer
            // and claim that barrier=stepover is a thing
            // would need substantial additional parsing of import data to fix it :(
            suspectedAnswerEnumFiles = suspectedAnswerEnumFiles.filter { "StileTypeAnswer.kt" !in it.name }
            return addedOrEditedTagsRealParsing(description, fileSourceCode, suspectedAnswerEnumFiles)
        }
        return null
    }

    @Serializable
    data class IncompleteCountryInfo(
        val additionalStreetsignLanguages: Set<String> = setOf(),
        val officialLanguages: Set<String> = setOf(),
    )

    private fun possibleLanguageKeys(): MutableSet<String> {
        val languageTags = mutableSetOf("name", "int_name")
        File(COUNTRY_METADATA_PATH_WITH_SLASH_ENDING).walkTopDown().maxDepth(1).forEach { file ->
            if (file.isFile) {
                val test = Yaml(configuration = YamlConfiguration(strictMode = false)).decodeFromString(IncompleteCountryInfo.serializer(), loadFileText(file))
                val langs = test.officialLanguages + test.additionalStreetsignLanguages
                // if country has single language for street names (langs.size = 1) then
                // it is using only name tag
                // if multiple languages are present then this languages are tagged with
                // name:$langCode tags, such as name:en for English language names
                if (langs.size > 1) {
                    // international counts for purposes of triggering multi-language support
                    // but itself is rather tagged with int_name tag and listed above already
                    langs.filter { it != "international" }.forEach { langCode ->
                        languageTags.add("name:$langCode")
                    }
                }
            }
        }
        return languageTags
    }

    private fun candidatesForEnumFilesForGivenFile(file: File): List<File> {
        val suspectedAnswerEnumFilesBasedOnFolder = candidatesForEnumFilesBasedOnFolder(file.parentFile)
        return suspectedAnswerEnumFilesBasedOnFolder + candidatesForEnumFilesBasedOnImports(file)
    }

    private fun candidatesForEnumFilesBasedOnFolder(folder: File): List<File> {
        val suspectedAnswerEnumFiles = mutableListOf<File>()
        File(folder.toString()).walkTopDown().forEach {
            if (isLikelyAnswerEnumFile(it)) {
                suspectedAnswerEnumFiles.add(it)
            }
        }
        return suspectedAnswerEnumFiles
    }

    private fun candidatesForEnumFilesBasedOnImports(file: File): List<File> {
        // initially just files from folder were taken as a base
        // due to cases like AddCrossing reaching across folders
        // it was not working well and require this extra parsing
        //
        // also, just parsing imports is not sufficient
        // see AddBikeParkingType which is not explicitly
        // importing the enum
        //
        // note: importedByFile may have false negatives that require extra parsing
        // to handle this
        return importedByFile(file)
            .filter { isLikelyAnswerEnumFile(File(it)) }
            .map { File(it) }
            .filter { it.isFile }
    }

    private fun importedByFile(file: File): Set<String> {
        val returned = mutableSetOf<String>()
        file.parse().locateByDescription("importList").forEach { importList ->
            importList.locateByDescription("importHeader").forEach {
                if (it is DefaultAstNode) {
                    areDirectChildrenMatchingStructureThrowExceptionIfNot("checking import file structure for $path", listOf(listOf("IMPORT", "WS", "identifier", "semi")), it, eraseWhitespace = false)
                    val imported = it.locateSingleOrExceptionByDescriptionDirectChild("identifier")
                    val identifier = imported.locateByDescriptionDirectChild("simpleIdentifier")
                    val pathsFromImportRoot = identifier.joinToString("/") { partBetweenDots ->
                        (partBetweenDots.tree() as KlassIdentifier).identifier
                    } + ".kt"
                    val importedPath = KOTLIN_IMPORT_ROOT_WITH_SLASH_ENDING + pathsFromImportRoot
                    if (File(importedPath).isFile) {
                        // TODO WARNING HACK: false positives here can be expected
                        // TODO WARNING HACK: this will treat
                        // import de.westnordost.streetcomplete.data.user.achievements.EditTypeAchievement.PEDESTRIAN
                        // as import of PEDESTRIAN.kt file
                        // not as import of PEDESTRIAN from EditTypeAchievement.kt file

                        // and this check will result in false negatives in turn...
                        returned.add(importedPath)
                    }
                }
            }
        }
        return returned
    }

    private fun isLikelyAnswerEnumFile(file: File): Boolean {
        // answers true if it is likely to contain an enum class like
        // java/de/westnordost/streetcomplete/quests/barrier_type/BarrierType.kt
        // contains
        //
        // TODO: maybe read file source code and simply check is "enum class" text there?
        if (".kt" !in file.name) {
            return false
        }
        val banned = listOf("SelectPuzzle.kt", "Form.kt", "Util.kt", "Utils.kt", "Adapter.kt",
            "Drawable.kt", "Dao.kt", "Dialog.kt", "Item.kt", "RotateContainer.kt")
        banned.forEach { if (it in file.name) {
            return false
        }
        }
        listOf("OsmFilterQuestType.kt", "MapDataWithGeometry.kt", "Element.kt", "Tags.kt",
            "OsmElementQuestType.kt", "CountryInfos.kt").forEach {
            if (it == file.name) {
                return false
            }
        }
        return !isQuestFile(file)
    }

    private fun addedOrEditedTagsRealParsing(description: String, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): Set<Tag>? {
        val ast = AstSource.String(description, fileSourceCode).parse()
        val defaultFunction = ast.extractFunctionByName(NAME_OF_FUNCTION_EDITING_TAGS)!!
        val functionSourceCode = defaultFunction.relatedSourceCode(fileSourceCode)
        if ("answer.applyTo(" !in functionSourceCode && "answer.litStatus.applyTo" !in functionSourceCode) {
            return addedOrEditedTagsWithGivenFunction(description, fileSourceCode, NAME_OF_FUNCTION_EDITING_TAGS, suspectedAnswerEnumFiles)
        } else {
            suspectedAnswerEnumFiles.forEach { fileHopefullyWithApplyTo ->
                val found = fileHopefullyWithApplyTo.parse().extractFunctionByName("applyTo")
                if (found != null) {
                    // OK, so we found related file providing applyTo function. Great!
                    if ("ParkingFee" in description) {
                        println("$description fpund apply to file $fileHopefullyWithApplyTo")
                    }
                    val got = addedOrEditedTagsRealParsingFindRealEditFunctionViaApplyToFunction(description, fileHopefullyWithApplyTo, suspectedAnswerEnumFiles)

                    val bonusScan = addedOrEditedTagsWithGivenFunction(description, fileSourceCode, NAME_OF_FUNCTION_EDITING_TAGS, suspectedAnswerEnumFiles)
                    if (bonusScan != null && bonusScan.isNotEmpty()) {
                        println(bonusScan)
                        throw ParsingInterpretationException("turns out to be needed")
                    }

                    if (got != null) {
                        return got
                    }
                }
            }
        }
        return null
    }

    private fun addedOrEditedTagsRealParsingFindRealEditFunctionViaApplyToFunction(description: String, fileWithRedirectedFunction: File, suspectedAnswerEnumFiles: List<File>): Set<Tag>? {
        val found = fileWithRedirectedFunction.parse().extractFunctionByName("applyTo")!!
        val parameters = found.locateSingleOrExceptionByDescriptionDirectChild("functionValueParameters")
            .locateByDescriptionDirectChild("functionValueParameter")
        if (parameters.isEmpty()) {
            throw ParsingInterpretationException("unsupported")
        }
        val parametersInCalledFunction = mutableListOf<String>()
        for (element in parameters) {
            val parameter = element.locateSingleOrExceptionByDescriptionDirectChild("parameter")
            val parameterTree = parameter.tree()
            if (parameterTree is KlassIdentifier) {
                parametersInCalledFunction.add(parameterTree.identifier)
            } else {
                throw ParsingInterpretationException("should not happen")
            }
        }
        if (parameters.size > 1) {
            println("Attempting to decompose function call where tags will be modified in another place and multiple arguments were passed")
            println("$description - parametersInCalledFunction in file ${fileWithRedirectedFunction.name} $parametersInCalledFunction")
            throw ParsingInterpretationException("No support yet")
        }
        return if (parametersInCalledFunction[0] == "tags") {
            val replacementFunctionName = "applyTo"
            val replacementSourceCode = loadFileText(fileWithRedirectedFunction)
            val replacementDescription = fileWithRedirectedFunction.toString()
            addedOrEditedTagsWithGivenFunction(replacementDescription, replacementSourceCode, replacementFunctionName, suspectedAnswerEnumFiles)
        } else {
            // variable is not really supported within called function
            throw ParsingInterpretationException("redirected function, not using tags variable - unsupported TODO, exiting")
        }
    }

    private fun addedOrEditedTagsWithGivenFunction(description: String, fileSourceCode: String, relevantFunctionName: String, suspectedAnswerEnumFiles: List<File>): Set<Tag>? {
        val ast = AstSource.String(description, fileSourceCode).parse()
        val relevantFunction = ast.extractFunctionByName(relevantFunctionName)
        if (relevantFunction == null) {
            println(description)
            println(fileSourceCode)
            throw ParsingInterpretationException("$relevantFunctionName missing in code provided via $description!")
        }
        val appliedTags = mutableSetOf<Tag>()
        var failedExtraction = false
        val got = extractCasesWhereTagsAreAccessedWithIndex(description, relevantFunction, fileSourceCode, suspectedAnswerEnumFiles)
        appliedTags += got

        appliedTags += extractCasesWhereTagsAreAccessedWithFunction(description, relevantFunction, fileSourceCode, suspectedAnswerEnumFiles)

        val tagsThatShouldBeMoreSpecific = appliedTags
            .filter { it.value == null && !freeformKey(it.key) && !streetCompleteIsReusingAnyValueProvidedByExistingTagging(description, it.key) }
        if (tagsThatShouldBeMoreSpecific.isNotEmpty()) {
            tagsThatShouldBeMoreSpecific.forEach { println(it) }
            println("$description found tags which are not freeform but have no specified values")
            failedExtraction = true
        }
        if (appliedTags.size == 0) {
            return null // parsing definitely failed
        }
        if (failedExtraction) {
            return null
        }
        return appliedTags
    }

    private fun extractTextFromHardcodedString(passedTextHolder: Ast): String? {
        var textHolder = passedTextHolder

        val plausibleText = textHolder.locateByDescription("stringLiteral")
        if (plausibleText.size == 1) {
            val textFoundIfFillingEntireHolder = plausibleText[0]
            if (textHolder.codeRange() == textFoundIfFillingEntireHolder.codeRange()) {
                // actual text holder is hidden inside, but it is actually the same object
                val expectedTextHolder = textFoundIfFillingEntireHolder.tree()
                if (expectedTextHolder is KlassString) {
                    textHolder = expectedTextHolder
                }
            }
        }
        if (textHolder is KlassString) {
            if (textHolder.children.size == 1) {
                val expectedText = textHolder.children[0]
                if (expectedText is StringComponentRaw) {
                    return expectedText.string
                }
            }
        }
        return null
    }

    private fun extractCasesWhereTagsAreAccessedWithIndex(description: String, relevantFunction: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): Set<Tag> {
        // it is trying to detect things like
        // tags["bollard"] = answer.osmValue
        val appliedTags = mutableSetOf<Tag>()
        relevantFunction.locateByDescription("assignment").forEach { assignment ->
            assignment.children.forEach { tagsDictAccess ->
                if (assignsToTagsVariable(tagsDictAccess)) {
                    // this limits it to things like
                    // tags[something] = somethingElse
                    // (would it also detect tags=whatever)?
                    val indexingElement = tagsDictAccess.locateSingleOrExceptionByDescription("assignableSuffix")
                        .locateSingleOrExceptionByDescription("indexingSuffix")
                    // indexingElement is something like ["indoor"] or [key]
                    val expression = indexingElement.locateSingleOrExceptionByDescriptionDirectChild("expression") // drop outer [ ]
                    val potentialTexts = expression.locateByDescription("stringLiteral", debug = false) // what if it is something like "prefix" + CONSTANT ?
                    val potentiallyUsableExpression = if (expression is KlassIdentifier) { expression } else { null } //
                    val likelyVariable = expression.locateByDescriptionDirectChild("disjunction") // tag[key] = ... for example
                    if (potentialTexts.size == 1) {
                        val processed = potentialTexts[0].tree()
                        if (processed == null) {
                            throw ParsingInterpretationException("not handled")
                        }
                        val key = extractTextFromHardcodedString(processed)
                        if (key == null) {
                            throw ParsingInterpretationException("key not found")
                        } else {
                            val valueHolder = assignment.locateSingleOrExceptionByDescriptionDirectChild("expression")
                            appliedTags += extractValuesForKnownKey(description, key, valueHolder, fileSourceCode, suspectedAnswerEnumFiles)
                        }
                    } else if (potentiallyUsableExpression != null) {
                        throw ParsingInterpretationException("expression in identified access as a variable")
                    } else if (likelyVariable.size == 1) {
                        if (likelyVariable[0].relatedSourceCode(fileSourceCode) == "key" && "name:\$languageTag" in fileSourceCode) {
                            // special handling for name quests
                            possibleLanguageKeys().forEach { appliedTags.add(Tag(it, null)) }
                        } else {
                            throw ParsingInterpretationException("expression in identified access as a complex variable")
                        }
                    } else {
                        throw ParsingInterpretationException("not handled, ${potentialTexts.size} texts, $potentiallyUsableExpression variable")
                    }
                }
            }
        }
        return appliedTags
    }

    private fun assignsToTagsVariable(tagsDictAccess: Ast): Boolean {
        return tagsDictAccess.description == "directlyAssignableExpression" &&
            tagsDictAccess is DefaultAstNode &&
            tagsDictAccess.children[0].tree() is KlassIdentifier &&
            ((tagsDictAccess.children[0].tree() as KlassIdentifier).identifier == "tags")
    }

    class EnumFieldState(val identifier: String, val possibleValue: String) {
        // entry such as
        // osmKey = building
        // from
        // HOUSE           ("building", "house"),
        // from
        // enum class BuildingType(val osmKey: String, val osmValue: String) {
        override fun toString(): String {
            return "EnumFieldState($identifier, $possibleValue)"
        }
    }

    class EnumEntry(val identifier: String, val fields: List<EnumFieldState>) {
        // entry such as
        // HOUSE           ("building", "house"),
        // from
        // enum class BuildingType(val osmKey: String, val osmValue: String) {
        override fun toString(): String {
            return "EnumEntry($identifier, $fields)"
        }
    }

    private fun getEnumValuesDefinedInThisFile(description: String, file: File, debug: Boolean = false): Set<EnumEntry> {
        val values = mutableSetOf<EnumEntry>()
        val fileMaybeContainingEnumSourceCode = loadFileText(file)
        val potentialEnumFileAst = file.parse()
        var enumsTried = 0
        potentialEnumFileAst.locateByDescription("classDeclaration").forEach { enum ->
            val modifiers = enum.locateByDescription("modifiers")
            if (modifiers.size != 1) {
                // not expected to be enum
                // will happen if potential enum file contains rather class such as
                // class StreetSideSelectRotateContainer @JvmOverloads constructor(
                return@forEach // skip silently as heuristic being too eager
            } else if (modifiers[0].relatedSourceCode(fileMaybeContainingEnumSourceCode) == "enum") {
                enumsTried += 1
                val enumFieldNames = mutableListOf<String>()
                val constructor = enum.locateSingleOrNullByDescription("primaryConstructor")
                if (constructor == null) {
                    // may happen with helper enums being present, such as
                    // enum class FireHydrantDiameterMeasurementUnit { MILLIMETER, INCH }
                    return@forEach // skip silently as heuristic being too eager
                }
                constructor.locateSingleOrExceptionByDescriptionDirectChild("classParameters")
                    .locateByDescriptionDirectChild("classParameter")
                    .forEach {
                        val simpleIdentifier = it.locateSingleOrExceptionByDescriptionDirectChild("simpleIdentifier")
                            .relatedSourceCode(fileMaybeContainingEnumSourceCode)
                        enumFieldNames.add(simpleIdentifier)
                    }
                enum.locateByDescription("enumEntry").forEach { enumEntry ->
                    var extractedText: String?
                    val identifier = (enumEntry.locateSingleOrNullByDescriptionDirectChild("simpleIdentifier")!!.tree() as KlassIdentifier).identifier
                    val valueArguments = enumEntry.locateSingleOrNullByDescriptionDirectChild("valueArguments")
                    if (valueArguments == null) {
                        val explanation = "parsing ${file.path} failed, valueArguments count is not 1, skipping, maybe it should be also investigated"
                        throw ParsingInterpretationException(explanation)
                    } else {
                        val enumFieldGroup = mutableListOf<EnumFieldState>()
                        val arguments = valueArguments.locateByDescriptionDirectChild("valueArgument")
                        for (i in arguments.indices) {
                            extractedText = extractTextFromHardcodedString(arguments[i])
                            if (extractedText == null) {
                                if (arguments[i].tree() is KlassDeclaration && (arguments[i].tree() as KlassDeclaration).identifier.toString() == "null") {
                                    // it has null as value, apparently
                                    // lest skip it silently
                                } else {
                                    throw ParsingInterpretationException("showHumanReadableTreeWithSourceCode(fileMaybeContainingEnumSourceCode) - showing ${file.path} after enum extraction failed")
                                }
                            } else {
                                enumFieldGroup.add(EnumFieldState(enumFieldNames[i], extractedText))
                            }
                        }
                        if (enumFieldGroup.size > 0) {
                            values.add(EnumEntry(identifier, enumFieldGroup))
                        }
                    }
                }
            }
        }
        if (values.size == 0 && debug) {
            println("enum extraction from ${file.path} failed! $enumsTried potential enums tried ($description request)")
        }
        return values
    }

    private fun extractValuesForKnownKey(description: String, key: String, valueHolder: Ast, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag> {
        val appliedTags = mutableSetOf<Tag>()

        var scanned: MutableSet<Tag>?
        scanned = extractValuesForKnownKeyFromWhenExpressionIfSingleOneIsPresent(description, key, valueHolder, fileSourceCode, suspectedAnswerEnumFiles)
        if (scanned != null) {
            return scanned
        }

        scanned = extractValuesForKnownKeyFromIfExpressionIfSingleOneIsPresent(description, key, valueHolder, fileSourceCode, suspectedAnswerEnumFiles)
        if (scanned != null) {
            return scanned
        }

        val valueIfItIsSimpleText = extractTextFromHardcodedString(valueHolder)
        val valueHolderSourceCode = valueHolder.relatedSourceCode(fileSourceCode)
        if (valueIfItIsSimpleText != null) {
            appliedTags.add(Tag(key, valueIfItIsSimpleText))
        } else if (valueHolderSourceCode.endsWith(".toYesNo()")) {
            appliedTags.add(Tag(key, "yes"))
            appliedTags.add(Tag(key, "no"))
        } else if (valueHolderSourceCode.endsWith(".toCheckDateString()")) {
            appliedTags.add(Tag(key, null))
        } else if (valueHolderSourceCode == "answer.joinToString(\";\") { it.osmValue }") { // answer.joinToString(";") { it.osmValue }
            val filtered = valueHolder.locateSingleOrExceptionByDescription("lambdaLiteral").locateSingleOrExceptionByDescriptionDirectChild("statements")
            appliedTags += provideTagsBasedOnAswerDataStructuresFromExternalFiles(description, key, filtered, suspectedAnswerEnumFiles)
            appliedTags.add(Tag(key, null)) // as it can be joined in basically any combination and listing all permutations would be absurd. Maybe provide comment here of taginfo listing supports this?
        } else if (valueHolderSourceCode.startsWith("answer.") || valueHolderSourceCode.startsWith("this.")) {
            appliedTags += provideTagsBasedOnAswerDataStructuresFromExternalFiles(description, key, valueHolder, suspectedAnswerEnumFiles)
        } else {
            if ( freeformKey(key) || streetCompleteIsReusingAnyValueProvidedByExistingTagging(description, key)) {
                appliedTags.add(Tag(key, null))
            } else {
                val explanation = "exact value is missing, extractValuesForKnownKey failed. $description get value (key is known: $key) from <$valueHolderSourceCode> somehow..."
                throw ParsingInterpretationException(explanation)
            }
        }
        return appliedTags
    }

    private fun provideTagsBasedOnAswerDataStructuresFromExternalFiles(description: String, key: String, valueHolder: Ast, suspectedAnswerEnumFiles: List<File>, debug: Boolean = false): MutableSet<Tag> {
        val appliedTags = mutableSetOf<Tag>()
        var extractedSomething = false
        suspectedAnswerEnumFiles.forEach {
            getEnumValuesDefinedInThisFile(description, it).forEach { enumGroup ->
                enumGroup.fields.forEach { value ->
                    // why redefined in each cycle?
                    // because there are cases where it would fail - but these are also cases
                    // where extracting enum also fails, so is not triggered and can be ignored
                    val postfixUnarySuffixes = valueHolder.locateByDescription("postfixUnarySuffix")
                    if (postfixUnarySuffixes.size != 1) {
                        throw ParsingInterpretationException("$key values extraction in provideTagsBasedOnAswerDataStructuresFromExternalFiles - postfixUnarySuffix expected to be a single one, got ${postfixUnarySuffixes.size}")
                    }
                    val accessIdentifierAst = postfixUnarySuffixes[0]
                        .locateSingleOrExceptionByDescriptionDirectChild("navigationSuffix")
                        .locateSingleOrExceptionByDescriptionDirectChild("simpleIdentifier")
                    val identifier = (accessIdentifierAst.tree() as KlassIdentifier).identifier

                    if (value.identifier == identifier) {
                        appliedTags.add(Tag(key, value.possibleValue))
                        extractedSomething = true
                        if (debug) {
                            println("$key=${value.possibleValue} registered based on ${value.identifier} identifier matching expected $identifier - from ${it.name}")
                        }
                    }
                }
            }
        }
        if (!freeformKey(key)) {
            // with freeform keys heuristic below will just get
            // variable such as capacity and will get confused
            // It is possible to get it working but not worth it right now
            suspectedAnswerEnumFiles.forEach { file ->
                val code = loadFileText(file)
                val ast = file.parse()
                val classDeclarations = ast.locateByDescription("classDeclaration")
                if (classDeclarations.isEmpty()) {
                    return@forEach
                }
                ast.locateByDescription("propertyDeclaration").forEach { astNode ->
                    val whenExpression = astNode.locateSingleOrNullByDescription("whenExpression")
                    if (whenExpression != null) {
                        extractValuesForKnownKeyFromWhenExpression(description, "dummykey", whenExpression, code, listOf()).forEach {
                            if (debug) {
                                println("OBTAINED FROM WHEN IN CLASS DECLARATION! $description $key=${it.value}")
                            }
                            appliedTags.add(Tag(key, it.value))
                            extractedSomething = true
                        }
                    }
                }
            }
        }

        if (!extractedSomething) {
            appliedTags.add(Tag(key, null))
            if ( freeformKey(key) || streetCompleteIsReusingAnyValueProvidedByExistingTagging(description, key)) {
                // no reason to complain
            } else {
                throw ParsingInterpretationException("failed to find values for now - key is $key")
            }
        }
        return appliedTags
    }

    private fun extractValuesForKnownKeyFromIfExpressionIfSingleOneIsPresent(description: String, key: String, valueHolder: Ast, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag>? {
        val ifExpression = valueHolder.locateSingleOrNullByDescription("ifExpression")
        if (ifExpression != null) {
            if (ifExpression.relatedSourceCode(fileSourceCode) == valueHolder.relatedSourceCode(fileSourceCode)) {
                return extractValuesForKnownKeyFromIfExpression(description, key, ifExpression, fileSourceCode, suspectedAnswerEnumFiles)
            } else {
                throw ParsingInterpretationException("not handled, when expressions as part of something bigger")
            }
        }
        return null
    }

    private fun extractValuesForKnownKeyFromIfExpression(description: String, key: String, ifExpression: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag> {
        val appliedTags = mutableSetOf<Tag>()
        ifExpression.locateByDescription("controlStructureBody").forEach {
            appliedTags += extractValuesForKnownKey(description, key, it, fileSourceCode, suspectedAnswerEnumFiles)
        }
        return appliedTags
    }

    private fun extractValuesForKnownKeyFromWhenExpressionIfSingleOneIsPresent(description: String, key: String, valueHolder: Ast, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag>? {
        val whenExpression = valueHolder.locateSingleOrNullByDescription("whenExpression")
        if (whenExpression != null) {
            if (whenExpression.relatedSourceCode(fileSourceCode) == valueHolder.relatedSourceCode(fileSourceCode)) {
                return extractValuesForKnownKeyFromWhenExpression(description, key, whenExpression, fileSourceCode, suspectedAnswerEnumFiles)
            } else {
                throw ParsingInterpretationException("not handled, when expressions as part of something bigger")
            }
        }
        return null
    }

    private fun extractValuesForKnownKeyFromWhenExpression(description: String, key: String, whenExpression: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag> {
        val appliedTags = mutableSetOf<Tag>()
        whenExpression.locateByDescription("whenEntry").forEach { it ->
            val structure = it.children.filter { it.description != "WS" }
            val expectedStructureA = listOf("whenCondition", "ARROW", "controlStructureBody", "semi")
            val expectedStructureB = listOf("ELSE", "ARROW", "controlStructureBody", "semi")
            areDirectChildrenMatchingStructureThrowExceptionIfNot(description, listOf(expectedStructureA, expectedStructureB), it, eraseWhitespace = true)
            appliedTags += extractValuesForKnownKey(description, key, structure[2], fileSourceCode, suspectedAnswerEnumFiles)
        }
        return appliedTags
    }

    private fun areDirectChildrenMatchingStructureThrowExceptionIfNot(description: String, expectedStructures: List<List<String>>, expression: AstNode, eraseWhitespace: Boolean) {
        val structure = expression.children.filter { !(eraseWhitespace && it.description == "WS") }.map { it.description }
        expectedStructures.forEach {
            if (it == structure) {
                return
            }
        }
        var maxLength = 0
        expectedStructures.forEach { if (maxLength < it.size) { maxLength = it.size } }
        for (i in 0 until maxLength) {
            expectedStructures.forEach {
                if (it.size > i) {
                    if (it[i] != structure[i]) {
                        throw ParsingInterpretationException("unexpected structure! at $i index")
                    }
                }
            }
        }
    }

    private fun surveyMarkKeyBasedOnKey(key: String): String {
        // TODO - can we directly call relevant StreetComplete code?
        return "$SURVEY_MARK_KEY:$key"
    }

    private fun extractCasesWhereTagsAreAccessedWithFunction(description: String, relevantFunction: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): Set<Tag> {
        // it is trying to detect things like
        // tags.updateWithCheckDate("smoking", answer.osmValue)
        val appliedTags = mutableSetOf<Tag>()
        relevantFunction.locateByDescription("postfixUnaryExpression")
            .filter { isAccessingTagsVariableWithMemberFunction(it) }
            .forEach { accessingTagsWithFunction ->
                val dotAndFunction = accessingTagsWithFunction.locateByDescriptionDirectChild("postfixUnarySuffix")[0].locateSingleOrExceptionByDescriptionDirectChild("navigationSuffix")

                if (dotAndFunction !is AstNode) {
                    throw ParsingInterpretationException("unexpected!")
                }
                val functionName = getNameOfFunctionFromNavigationSuffix(dotAndFunction)
                if (functionName in listOf(
                        "setCheckDateForKey",
                        "updateCheckDateForKey"
                    )
                ) {
                    // only check data for
                    val keyString = extractStringLiteralArgumentInFunctionCall(description, 0, accessingTagsWithFunction)
                    if (keyString != null) {
                        appliedTags.add(Tag(surveyMarkKeyBasedOnKey(keyString), null))
                    }
                } else if (functionName ==  "updateWithCheckDate") {
                    var keyString = extractStringLiteralArgumentInFunctionCall(description, 0, accessingTagsWithFunction)
                    val valueString = extractStringLiteralArgumentInFunctionCall(description, 1, accessingTagsWithFunction)

                    // fold it into extractArgumentInFunctionCall?
                    // try to automatically obtain this constants?
                    if (keyString == null) {
                        val keyArgumentAst = extractArgumentSyntaxTreeInFunctionCall(0, accessingTagsWithFunction).locateSingleOrNullByDescription("primaryExpression")
                        if (keyArgumentAst == null) {
                            throw ParsingInterpretationException("unexpected")
                        }
                        val keyArgumentAstTree = keyArgumentAst.tree()
                        if (keyArgumentAstTree is KlassIdentifier) {
                            if (keyArgumentAstTree.identifier == "SOUND_SIGNALS") {
                                keyString = SOUND_SIGNALS
                            }
                            if (keyArgumentAstTree.identifier == "VIBRATING_BUTTON") {
                                keyString = VIBRATING_BUTTON
                            }
                        }
                    }

                    if (keyString != null) {
                        appliedTags.add(Tag("$SURVEY_MARK_KEY:$keyString", null))
                        if (valueString != null) {
                            appliedTags.add(Tag(keyString, valueString))
                        } else {
                            val valueAst = extractArgumentSyntaxTreeInFunctionCall(1, accessingTagsWithFunction)
                            val valueHolderSourceCode = valueAst.relatedSourceCode(fileSourceCode)
                            if (valueHolderSourceCode == "answer.toYesNo()") {
                                // kind of hackish, fix this?
                                appliedTags.add(Tag(keyString, "yes"))
                                appliedTags.add(Tag(keyString, "no"))
                            } else if (valueHolderSourceCode == "answer.osmValue" || valueHolderSourceCode == "answer.value.osmValue") {
                                val dotAcess = valueAst.locateByDescription("postfixUnarySuffix")
                                if (dotAcess.isEmpty()) {
                                    throw ParsingInterpretationException("hmmmmmmmm")
                                }
                                val accessIdentifierAst = dotAcess[dotAcess.size - 1].locateSingleOrExceptionByDescriptionDirectChild("navigationSuffix")
                                    .locateSingleOrExceptionByDescriptionDirectChild("simpleIdentifier")
                                val identifier = (accessIdentifierAst.tree() as KlassIdentifier).identifier
                                var extractedNothing = true
                                suspectedAnswerEnumFiles.forEach {
                                    getEnumValuesDefinedInThisFile(description, it).forEach { value ->
                                        // dotAcess will have a single element [.osmValue] on "answer.osmValue"
                                        // dotAcess will have a two elements [.value, .osmValue] on "answer.value.osmValue"
                                        if (value.fields.size != 1) {
                                            throw ParsingInterpretationException("expected a single value, got $value")
                                        }
                                        if (value.fields[0].identifier == identifier) {
                                            appliedTags.add(Tag(keyString, value.fields[0].possibleValue))
                                        }
                                        extractedNothing = false
                                    }
                                }
                                if (extractedNothing) {
                                    throw ParsingInterpretationException("Enum obtaining failed! suspectedAnswerEnumFiles $suspectedAnswerEnumFiles")
                                }
                            } else {
                                val valueSourceCode = valueAst.relatedSourceCode(fileSourceCode)
                                if (freeformKey(keyString) && valueSourceCode in setOf("answer.toString()", "openingHoursString", "answer.times.toString()", "duration.toOsmValue()", "toOsmValue()")) {
                                    // key is freeform and it appears to not be enum - so lets skip complaining and attempting to tarck down value
                                    // individual quests can be investigated as needed
                                    appliedTags.add(Tag(keyString, null))
                                } else {
                                    throw ParsingInterpretationException("extractCasesWhereTagsAreAccessedWithFunction - extraction failing: $description tags dict is accessed with updateWithCheckDate, key known ($keyString), value unknown, obtaining data failed")
                                }
                            }
                        }
                    } else {
                        throw ParsingInterpretationException("$description - failed to extract key from updateWithCheckDate")
                    }
                } else if (functionName in listOf("remove", "containsKey", "removeCheckDatesForKey", "hasChanges", "entries", "hasCheckDateForKey", "hasCheckDate")) {
                    // skip, as only added or edited tags are listed - and removed one and influencing ones are ignored
                } else if (functionName in listOf("updateCheckDate")) {
                    appliedTags.add(Tag(SURVEY_MARK_KEY, null))
                } else if (functionName == "replaceShop") {
                    // parsing skipped per
                    // https://github.com/streetcomplete/StreetComplete/issues/4225#issuecomment-1190487094
                } else {
                    throw ParsingInterpretationException("unexpected function name $functionName in $description")
                }
            }
        return appliedTags
    }

    private fun isAccessingTagsVariableWithMemberFunction(ast: AstNode): Boolean {
        val root = ast.tree()
        if (root !is KlassIdentifier) {
            return false
        }
        if (root.identifier != "tags") {
            return false
        }
        val primary = ast.locateSingleOrExceptionByDescriptionDirectChild("primaryExpression")
        val rootOfExpectedTagsIdentifier = primary.tree()
        if (rootOfExpectedTagsIdentifier !is KlassIdentifier) {
            throw ParsingInterpretationException("unexpected! primary is ${primary::class}")
        }
        if (rootOfExpectedTagsIdentifier.identifier != "tags") {
            throw ParsingInterpretationException("unexpected!")
        }
        val possibleDotAndFunction = ast.locateByDescriptionDirectChild("postfixUnarySuffix")
        if (possibleDotAndFunction.isEmpty()) {
            // this will happen in case of say
            // tags["key"] = value
            // in such case we want to skip it
            return false
        }
        val expectedToHoldDotAndFunctionCall = possibleDotAndFunction[0].locateByDescriptionDirectChild("navigationSuffix")
        if (expectedToHoldDotAndFunctionCall.isEmpty()) {
            // maybe false positive?
            // maybe something like
            // .any { tags[it]?.toCheckDate() != null }
            // where skipping is valid?
            return false
        }
        return true
    }

    private fun extractArgumentListSyntaxTreeInFunctionCall(ast: AstNode): List<AstNode> {
        val arguments = ast.locateByDescriptionDirectChild("postfixUnarySuffix")[1]
            .locateSingleOrExceptionByDescriptionDirectChild("callSuffix")
            .locateSingleOrExceptionByDescriptionDirectChild("valueArguments")
        return arguments.locateByDescription("valueArgument")
    }

    private fun extractArgumentSyntaxTreeInFunctionCall(index: Int, ast: AstNode): AstNode {
        return extractArgumentListSyntaxTreeInFunctionCall(ast)[index]
    }

    private fun extractStringLiteralArgumentInFunctionCall(description: String, index: Int, ast: AstNode): String? {
        val found = extractArgumentSyntaxTreeInFunctionCall(index, ast).locateSingleOrNullByDescription("primaryExpression")
        if (found == null) {
            throw ParsingInterpretationException("war")
        }
        if (found.children.size == 1) {
            return if (found.children[0].description == "stringLiteral") {
                val stringObject = (found.children[0].tree() as KlassString).children[0]
                (stringObject as StringComponentRaw).string
            } else {
                // as function name mentions, only string literals will be recovered
                null
            }
        } else {
            throw ParsingInterpretationException("unhandled extraction of $index function parameter - multiple children")
        }
    }

    private fun getNameOfFunctionFromNavigationSuffix(dotAndFunction: AstNode): String {
        if (dotAndFunction.description != "navigationSuffix") {
            exitProcess(1)
        }
        val expectedPackagedDot = dotAndFunction.children[0]
        if (expectedPackagedDot.description != "memberAccessOperator") {
            throw ParsingInterpretationException("unexpected!")
        }
        if (expectedPackagedDot !is AstNode) {
            throw ParsingInterpretationException("unexpected!")
        }
        val expectedDot = expectedPackagedDot.children[0]
        if (expectedDot !is DefaultAstTerminal) {
            throw ParsingInterpretationException("unexpected!")
        }
        if (expectedDot.text != ".") {
            throw ParsingInterpretationException("unexpected!")
        }
        val expectedFunctionIdentifier = dotAndFunction.children[1]
        if (expectedFunctionIdentifier.description != "simpleIdentifier") {
            throw ParsingInterpretationException("unexpected!")
        }
        if (expectedFunctionIdentifier.tree() !is KlassIdentifier) {
            throw ParsingInterpretationException("unexpected! expectedFunctionIdentifier.root() is ${expectedFunctionIdentifier.tree()!!::class}")
        }
        return (expectedFunctionIdentifier.tree() as KlassIdentifier).identifier
    }

    class ParsingInterpretationException(private val s: String) : Throwable() {
        override fun toString(): String {
            return s
        }
    }

    private fun Ast.codeRange(): Pair<Int, Int> {
        val start = tree()!!.astInfoOrNull!!.start.index
        val end = tree()!!.astInfoOrNull!!.stop.index
        return Pair(start, end)
    }

    private fun Ast.relatedSourceCode(sourceCode: String): String {
        if (tree() == null) {
            return "<source code not available>"
        }
        val start = tree()!!.astInfoOrNull!!.start.index
        val end = tree()!!.astInfoOrNull!!.stop.index
        if (start < 0 || end < 0) {
            return "<source code not available> - stated range was $start to $end index"
        }
        return sourceCode.subSequence(start, end).toString()
    }

    private fun Ast.locateSingleOrNullByDescription(filter: String, debug: Boolean = false): AstNode? {
        val found = locateByDescription(filter, debug)
        return if (found.size != 1) {
            null
        } else {
            found[0]
        }
    }

    private fun Ast.locateSingleOrExceptionByDescription(filter: String, debug: Boolean = false): AstNode {
        val found = locateByDescription(filter, debug)
        if (found.size != 1) {
            throw ParsingInterpretationException("unexpected count! Expected single matching on filter $filter, got ${found.size}")
        } else {
            return found[0]
        }
    }

    private fun Ast.locateByDescription(filter: String, debug: Boolean = false): List<AstNode> {
        if (this is AstNode) {
            val fromChildren = children.flatMap { child ->
                child.locateByDescription(filter, debug)
            }
            return if (description == filter) {
                if (debug) {
                    println("$filter filter matching description")
                }
                listOf(this) + fromChildren
            } else {
                if (debug) {
                    println("$filter filter NOT matching description $description")
                }
                fromChildren
            }
        } else {
            return listOf()
        }
    }

    private fun Ast.locateSingleOrExceptionByDescriptionDirectChild(filter: String): Ast {
        val found = locateByDescriptionDirectChild(filter)
        if (found.size != 1) {
            throw ParsingInterpretationException("unexpected count! Expected single matching direct child on filter $filter, got ${found.size}")
        } else {
            return found[0]
        }
    }

    private fun Ast.locateSingleOrNullByDescriptionDirectChild(filter: String): Ast? {
        val found = locateByDescriptionDirectChild(filter)
        return if (found.size != 1) {
            null
        } else {
            found[0]
        }
    }

    private fun Ast.locateByDescriptionDirectChild(filter: String): List<Ast> {
        val returned = mutableListOf<Ast>()
        if (this is AstNode) {
            for (child in children) {
                if (child.description == filter) {
                    returned.add(child)
                }
            }
        }
        return returned
    }

    private fun Ast.extractFunctionByName(functionName: String): AstNode? {
        val got = extractAllFunctionsByName(functionName)
        if (got.size > 1) {
            throw ParsingInterpretationException("expected one function, got multiple")
        }
        if (got.isEmpty()) {
            return null
        }
        return got[0]
    }

    private fun Ast.extractAllFunctionsByName(functionName: String): List<AstNode> {
        if (description == "functionDeclaration") {
            if (this is AstNode) {
                children.forEach {
                    if (it.description == "simpleIdentifier" && it.tree() is KlassIdentifier && ((it.tree() as KlassIdentifier).identifier == functionName)) {
                        return listOf(this) + children.flatMap { child ->
                            child.extractAllFunctionsByName(functionName)
                        }
                    }
                }
            } else {
                throw ParsingInterpretationException("wat")
            }
        }
        return if (this is AstNode) {
            children.flatMap { child ->
                child.extractAllFunctionsByName(functionName)
            }
        } else {
            listOf()
        }
    }

    private fun Ast.tree(): Ast? {
        var returned: Ast? = null
        this.summary(false).onSuccess { returned = it.firstOrNull() }
        return returned
    }

    private fun AstSource.parse() = KotlinGrammarAntlrKotlinParser.parseKotlinFile(this)

    private fun File.parse(): Ast {
        val inputStream: InputStream = this.inputStream()
        val text = inputStream.bufferedReader().use { it.readText() }
        return KotlinGrammarAntlrKotlinParser.parseKotlinFile(AstSource.String(this.path, text))
    }
}

\$\endgroup\$
1

1 Answer 1

2
\$\begingroup\$

The code is indeed very long, so just a few things I noticed while looking at it:

  • Overall, the code should be split into a lot more smaller classes with fewer responsibilities. Some methods are also way too long.
  • When working with streams, use use pattern for easier and correct closing/handling.
  • Do not throw empty Exception(), always provide message. Take a look at functions like error, require or check
  • You are OFTEN using .forEach and then inside you have a condition (or more) to manually filter your iterable. Instead, use .filter to filter things and then chain .forEach afterwards.
  • You have big methods with a lot of ifs. Changing them to when will make it easier. Even better would be to change into polymorphism when viable.
  • Class Tag (maybe other classes too) has overriden equals and hashcode when it could just be data class and have those autogenerated.
  • Use emptySet instead of setOf()
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.