1
\$\begingroup\$

I have a below struct where I have a nested map for CustomersIndex which allocates bunch of internal maps causing memory increase. I profiled it so I noticed this. I am trying to see if there is any way to redesign my CustomersIndex data structure which doesn't uses nested map?

const (
    departmentsKey = "departments"
)

type CustomerManifest struct {
    Customers      []definitions.Customer
    CustomersIndex map[int]map[int]definitions.Customer
}

This is the way it is being populated here in my below code:

func updateData(mdmCache *mdm.Cache) map[string]interface{} {
    memCache := mdmCache.MemCache()

    var customers []definitions.Customer
    var customersIndex = map[int]map[int]definitions.Customer{}

    for _, r := range memCache.Customer {
        customer := definitions.Customer{
            Id:           int(r.Id),
            SetId:        int(r.DepartmentSetId),
        }

        customers = append(customers, customer)
        _, yes := customersIndex[customer.SetId]
        if !yes {
            customersIndex[customer.SetId] = make(map[int]definitions.Customer)
        }
        customersIndex[customer.SetId][customer.Id] = customer
    }

    return map[string]interface{}{
        departmentsKey: &CustomerManifest{Customers: customers, CustomersIndex: customersIndex},
    }
}

And this is the way I am getting my CustomersIndex nested map.

func (c *Client) GetCustomerIndex() map[int]map[int]definitions.Customer {
    c.mutex.RLock()
    defer c.mutex.RUnlock()
    customersIndex := c.data[departmentsKey].(*CustomerManifest).CustomersIndex
    return customersIndex
}

Is there any way to design my CustomersIndex in a way where I don't have to use nested map? Also can nested map cause more allocations causing more memory increase?

\$\endgroup\$
1
  • 2
    \$\begingroup\$ Please change the title to indicate what the code is used for. That will help us review the code. Please read How do I ask a good question?. \$\endgroup\$
    – pacmaninbw
    Jun 25, 2022 at 16:00

1 Answer 1

1
\$\begingroup\$

I presume the definitions.Customer struct has more in it than just the Id and SetId fields? For example, a Name or other fields. (Otherwise the index wouldn't be useful.)

It may help to use a single map with a composite key. For example, define a key type with two int fields and index it by customer ID and department ("set ID"):

type indexKey struct {
    id    int
    setId int
}

var CustomersIndex map[indexKey]Customer

// ...
fmt.Println(CustomersIndex[indexKey{id: 1, setId: 2}])

Full example in Go Playground.

I suspect this will save memory to use a single map, but I'm not 100% sure. You'd need to compare the profiling results for your data.

Is it actually using too much memory the original way, or is this a theoretical concern?

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.