How to avoid nested map allocations in below code?

Question

I have a below struct where I have a nested map for CustomersIndex which allocates bunch of internal maps causing memory increase. I profiled it so I noticed this. I am trying to see if there is any way to redesign my CustomersIndex data structure which doesn't uses nested map?

const (
    departmentsKey = "departments"
)

type CustomerManifest struct {
    Customers      []definitions.Customer
    CustomersIndex map[int]map[int]definitions.Customer
}

This is the way it is being populated here in my below code:

func updateData(mdmCache *mdm.Cache) map[string]interface{} {
    memCache := mdmCache.MemCache()

    var customers []definitions.Customer
    var customersIndex = map[int]map[int]definitions.Customer{}

    for _, r := range memCache.Customer {
        customer := definitions.Customer{
            Id:           int(r.Id),
            SetId:        int(r.DepartmentSetId),
        }

        customers = append(customers, customer)
        _, yes := customersIndex[customer.SetId]
        if !yes {
            customersIndex[customer.SetId] = make(map[int]definitions.Customer)
        }
        customersIndex[customer.SetId][customer.Id] = customer
    }

    return map[string]interface{}{
        departmentsKey: &CustomerManifest{Customers: customers, CustomersIndex: customersIndex},
    }
}

And this is the way I am getting my CustomersIndex nested map.

func (c *Client) GetCustomerIndex() map[int]map[int]definitions.Customer {
    c.mutex.RLock()
    defer c.mutex.RUnlock()
    customersIndex := c.data[departmentsKey].(*CustomerManifest).CustomersIndex
    return customersIndex
}

Is there any way to design my CustomersIndex in a way where I don't have to use nested map? Also can nested map cause more allocations causing more memory increase?

Answer 1

I presume the definitions.Customer struct has more in it than just the Id and SetId fields? For example, a Name or other fields. (Otherwise the index wouldn't be useful.)

It may help to use a single map with a composite key. For example, define a key type with two int fields and index it by customer ID and department ("set ID"):

type indexKey struct {
    id    int
    setId int
}

var CustomersIndex map[indexKey]Customer

// ...
fmt.Println(CustomersIndex[indexKey{id: 1, setId: 2}])

Full example in Go Playground.

I suspect this will save memory to use a single map, but I'm not 100% sure. You'd need to compare the profiling results for your data.

Is it actually using too much memory the original way, or is this a theoretical concern?