F# Remove Duplicates From List (Seq.distinct)

Invoke Seq.distinct to remove duplicates from a list. Convert from the seq back into a list or array.

Remove duplicates. A list contains numbers: 10, 20, 10, 30. There are two tens. But we only want distinct values in this list. With F# we can eliminate dupes.

With Seq functions, we can apply common operations to lists. The distinct function receives a list. It eliminates the second occurrence of values in the list.

An example. Let us begin. We have a list of popular drinks. But there are two "coffee" strings in the list. We then call Seq.distinct and pass the list to it.

Result: The Seq.distinct function returns a seq instance. The second "coffee" was removed, and no other changes were made.

Seq

Finally: We use Seq.toList and Seq.toArray to convert our sequence to other types. These may be more useful in other parts of the program.

F# program that uses distinct, removes duplicates // This list has a duplicate string. let drinks = ["coffee"; "beer"; "tea"; "coffee"; "milk"] // Use Seq.distinct to remove duplicates. // ... The order is retained. let result = Seq.distinct drinks printfn "%A" result // We can convert from the Seq to a list or array. let resultList = Seq.toList result let resultArray = Seq.toArray result printfn "%A" resultList.Length printfn "%A" resultArray.Length Output seq ["coffee"; "beer"; "tea"; "milk"] 4 4

DistinctBy. Sometimes duplicate-removing (dedupe) can be augmented with transformations. For example we can trim whitespace from strings before comparing them.

Here: We use distinctBy and pass it a lambda expression. The lambda returns a trimmed string by calling Trim.

Fun

Result: The seq has no duplicates. The leading and trailing whitespace is ignored by distinctBy.

F# program that uses distinctBy, ignores whitespace // Has some duplicates if we ignore whitespace. let codes = [" ABC "; "DEF"; "ABC"; "XYZ"; " XYZ"] // Use Seq.distinctBy to trim strings before removing duplicates. let result = Seq.distinctBy (fun (a : string) -> a.Trim()) codes printfn "%A" result Output seq [" ABC "; "DEF"; "XYZ"]

Count by value. This function counts elements by value—if a sequence has two "Red" values it returns 2. It uses "where" to create an intermediate sequence with only matching elements.

And: With the pipeline operator, it returns the length of the sequence, which is the count of matching elements in the sequence.

Tip: This does not remove duplicates. But it tells us whether duplicates are present for a certain value.

F# program that counts instances in list // This count function accepts two arguments: value and list. // ... It computes a Seq where the value equals the argument. // It returns the length of that sequence. let count value list = Seq.where (fun y -> y = value) list |> Seq.length // Has two red values. let colors = ["Blue"; "Red"; "Red"; "Orange"] // Count blue values. let total1 = count "Blue" colors printfn "Blue = %A" total1 // Count red values. let total2 = count "Red" colors printfn "Red = %A" total2 Output Blue = 1 Red = 2

Int array. This program uses a pipelined expression to remove duplicate elements from an Int array. The result is a new Int array.Array

Description: We use Seq.ofArray to treat the array as a sequence. We then call distinct, and then toArray to return an array.

F# program that removes duplicates from Int array let numbers = [|10; 10; 20; 30; 40; 40|] // Use ofArray to use an array as a sequence. // ... Use distinct to eliminate duplicates. // Convert back into an Int array. let result = Seq.ofArray numbers |> Seq.distinct |> Seq.toArray printfn "%A" result Output [|10; 20; 30; 40|]

A review. With sequences in F#, we use prebuilt logic to process data. And we can combine operations into more powerful, sequence-based ones. This is powerful. It is elegant.

The Dev Codes

.Net

.NET Array Dictionary List String 2D Async DataTable Dates DateTime Enum File For Foreach Format IEnumerable If IndexOf Lambda LINQ Parse Path Process Property Regex Replace Sort Split Static StringBuilder Substring Switch Tuple

Java

Core Array ArrayList HashMap String 2D Cast Character Console Deque Duplicates File For Format HashSet If IndexOf Lambda Math ParseInt Process Random Regex Replace Sort Split StringBuilder Substring Switch Vector While

TheDeveloperBlog.com

F# Remove Duplicates From List (Seq.distinct)

Related Links:

.Net

Java

Related Links