1
\$\begingroup\$

Problem

I need to be able to grab a subset of characters but ensure that each set doesn't exceed a specified limit and must end with a defined set of characters. For example, say I have an entire book worth of characters, I need to chunk that into a set of 300 characters but ensure that none of those chunks is split mid-sentence.

To put it into pesudo code I basically need:

foreach(var chunk in text.Chunk(x).ButAlsoSplitOnFirst('.', '!', '?', etc...);

My solution

public static class StringExtensions
    {
        public static IEnumerable<string> ChunkWithDelimeters(this string source, int maxChar, params char[] delimeters)
        {
            var sourceSpan = source.AsSpan();
            var items = new List<string>();

            var pointer = 0;
            var lazyPointer = 0;

            while (pointer <= sourceSpan.Length)
            {
                bool foundMatch = false;

                pointer = Math.Min(lazyPointer + maxChar, sourceSpan.Length);

                if (pointer == sourceSpan.Length)
                {
                    items.Add(sourceSpan.Slice(lazyPointer, pointer - lazyPointer).TrimStart().ToString());
                    break;
                }

                for (var j = pointer; j >= lazyPointer; j--)
                {
                    var tempChar = sourceSpan[j];

                    if (delimeters.Contains(tempChar))
                    {
                        foundMatch = true;
                        items.Add(sourceSpan.Slice(lazyPointer, j + 1 - lazyPointer).TrimStart().ToString());
                        lazyPointer = j + 1;
                    }
                }

                if (!foundMatch)
                {
                    items.Add(sourceSpan.Slice(lazyPointer, pointer - lazyPointer).Trim().ToString());
                    lazyPointer = pointer;
                }
            }

            return items;
        }
    }

Benchmarks

source.Length maxChar delimiters Mean (μs) Error (μs) StdDev (μs)
7320000 30 '.' 73,622.739 1,412.4035 1,252.0589
439200 30 '.' 4,122.574 68.6392 60.8468
731 30 '.' 5.125 0.0466 0.0436

My question

What is the optimal solution to this problem? Is there any way I can improve the performance of my solution?

\$\endgroup\$
2
  • \$\begingroup\$ What if the sentence is longer than 300 chars? Which rule wins? \$\endgroup\$ – CharlesNRice Mar 18 at 14:54
  • \$\begingroup\$ Ensure that each set doesn't exceed a specified limit and must end with a defined set of characters unless those characters are not in the chunk? \$\endgroup\$ – Chandler Turner Mar 18 at 14:56
0
\$\begingroup\$

Don't you need to test for delimiters here? From reading your code I assume we want to max chunk but if inside the chunk there are delimiters we want to break that up. If the last section fits in the entire chunk it's not checking for delimiters

if (pointer == sourceSpan.Length)
{
    items.Add(sourceSpan.Slice(lazyPointer, pointer - lazyPointer).TrimStart().ToString());
    break;
}

Also you could use IndexOfAny instead of the loop at the end

WARNING this code not tested and just used an example

var slice = sourceSpan.Slice(lazyPointer, pointer - lazyPointer);
var contains = slice.IndexOfAny(delimeters);
while (contains > -1)
{
    items.Add(slice.Slice(0, contains + 1).ToString());
    slice = slice.Slice(contains + 1);
    contains = slice.IndexOfAny(delimeters);
}

Now some other points that I think you should split by delimiters first then chunks. Instead of chunks then delimiters. If I have a string of "12345.6789.8.654321" and split by 7. The first two coming out would be "12345" <-- looks good but then the next chunk would be "6" then chunk after that would be "789","8","6","54321" if we split by delimiters first then chunked them it would output what I think most people would expect.

\$\endgroup\$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.